An analysis of published trials found that current use of pragmatic trial labels is uninformative.

OBJECTIVES: Randomized trials labelled as "pragmatic" are attractive to funders, patients, and clinicians as the label implies that the results are directly applicable to clinical care. We examined how authors justify use of the label (e.g., by referring to one or more PRECIS [PRagmatic Explanatory Continuum Indicator Summary]-2 domains). STUDY DESIGN AND SETTING: We reviewed primary trial reports published 2014-2019, registered in ClinicalTrials.gov and using the pragmatic label anywhere in the report. RESULTS: Among 415 trials, the label was justified by reference to at least one design element in 282 (68.0%); of these, 240 (85.1%) referenced trial characteristics that can be mapped to one or more of the PRECIS-2 domains, most commonly eligibility (91, 32.3%), setting (90, 31.9%), flexibility delivery (89, 31.6%), and organization (75, 26.6%); 42 (14.9%) referenced characteristics that are not PRECIS-2 domains, most commonly type of intervention/comparator (48, 17%), recruitment without consent (22, 7.8%), routinely collected data (22, 7.8%), and cluster randomization (20, 7.1%). Most reports referenced only one or two design elements. Overall, 9/415 (2.2%) provided PRECIS wheels. CONCLUSION: Current use of pragmatic labels is uninformative. Authors should clarify the decision the trial is intended to support and include a PRECIS-2 table to make the design transparent.

Principal investigators over-optimistically forecast scientific and operational outcomes for clinical trials.

OBJECTIVE: To assess the accuracy of principal investigators' (PIs) predictions about three events for their own clinical trials: positivity on trial primary outcomes, successful recruitment and timely trial completion. STUDY DESIGN AND SETTING: A short, electronic survey was used to elicit subjective probabilities within seven months of trial registration. When trial results became available, prediction skill was calculated using Brier scores (BS) and compared against uninformative prediction (i.e. predicting 50% all of the time). RESULTS: 740 PIs returned surveys (16.7% response rate). Predictions on all three events tended to exceed observed event frequency. Averaged PI skill did not surpass uninformative predictions (e.g., BS = 0.25) for primary outcomes (BS = 0.25, 95% CI 0.20, 0.30) and were significantly worse for recruitment and timeline predictions (BS 0.38, 95% CI 0.33, 0.42; BS = 0.52, 95% CI 0.50, 0.55, respectively). PIs showed poor calibration for primary outcome, recruitment, and timelines (calibration index = 0.064, 0.150 and 0.406, respectively), modest discrimination in primary outcome predictions (AUC = 0.76, 95% CI 0.65, 0.85) but minimal discrimination in the other two outcomes (AUC = 0.64, 95% CI 0.57, 0.70; and 0.55, 95% CI 0.47, 0.62, respectively). CONCLUSION: PIs showed overconfidence in favorable outcomes and exhibited limited skill in predicting scientific or operational outcomes for their own trials. They nevertheless showed modest ability to discriminate between positive and non-positive trial outcomes. Low survey response rates may limit generalizability.

Evaluating the evidence behind the surrogate measures included in the FDA's table of surrogate endpoints as supporting approval of cancer drugs.

BACKGROUND: In July 2018, the FDA first published a table listing all surrogate measures that it has used, and may accept for future use, in regulatory approval. However, the strength of surrogacy for those measures was not formally assessed. Using the case example of breast cancer, we aimed to evaluate the strength of correlation of surrogate measures listed in the FDA's Table with overall survival. METHODS: This cross-sectional study of the FDA's Table of Surrogate Endpoints was conducted in May 2019. All surrogate measures listed in the FDA table as appropriate for accelerated or regular approval for breast cancer were extracted. We identified studies evaluating the correlation of treatment benefit in the surrogate with treatment benefit in overall survival and extracted results from the correlation analysis. FINDINGS: Five surrogate endpoints were listed for breast cancer in the FDA website: pathological complete response rates (pCR), event-free survival (EFS), disease-free survival (DFS), objective response rates (ORR), and progression-free survival (PFS), of which pCR was listed as appropriate only for accelerated approval, while the rest were considered appropriate for accelerated or regular approval. No correlation study evaluated the correlation of treatment effects on EFS with that on OS. The results from correlation studies evaluating pCR, DFS, ORR, and PFS suggest that the treatment effects on none of these surrogate measures were strongly correlated with treatment effects on OS (r<0.85 or R2  < 0.7, except for DFS in HER2 positive early breast cancer (R2  = 0.75). INTERPRETATION: Using breast cancer as an example, we evaluated the underlying evidence for the surrogate endpoints for solid tumors listed in the FDA's Table of Surrogate Endpoints and found weak or missing correlations of treatment effects on these surrogates with treatment effects on OS . Surrogate measures should be predictive of clinical benefit to be useful in supporting regular FDA approval. FUNDING: Work on this project was funded by the Arnold Ventures. Dr. Kesselheim is also supported by the Harvard-MIT Center for Regulatory Science. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

A search filter to identify pragmatic trials in MEDLINE was highly specific but lacked sensitivity.

OBJECTIVES: Identifying pragmatic trials from among all randomized trials is challenging because of inconsistent reporting. Our objective was to develop and validate a search filter to identify reports of pragmatic trials from Ovid MEDLINE. STUDY DESIGN AND SETTING: Two sets of known and probable pragmatic trial records were analyzed using text mining to generate candidate terms. Two large population sets comprising clinical trials and explanatory trials were used to select discriminating terms. Various combinations of terms were tested iteratively to achieve optimal search performance. Two externally derived sets were used to validate sensitivity and specificity of the derived filters. RESULTS: Our validated sensitivity-maximizing filter (combines trial design terms with terms relating to attributes of pragmatic trials) retrieves over 42,000 records in MEDLINE and has sensitivity of 46.4% (95% confidence interval (CI) 37.2 to 55.7%) and estimated specificity of 98.1% (95% CI 93.4 to 99.8%). Search performance is superior to other ad hoc filters for pragmatic trials. The Cochrane search for randomized trials has much better sensitivity (98.2%), but poorer specificity (1.9%) and retrieves 4.5 million records. CONCLUSION: A highly specific filter (low false positive rate) with moderate sensitivity is available for identifying reports of trials more likely to be pragmatic.

The ethical challenges raised in the design and conduct of pragmatic trials: an interview study with key stakeholders.

BACKGROUND: There is a concern that the apparent effectiveness of interventions tested in clinical trials may not be an accurate reflection of their actual effectiveness in usual practice. Pragmatic randomized controlled trials (RCTs) are designed with the intent of addressing this discrepancy. While pragmatic RCTs may increase the relevance of research findings to practice they may also raise new ethical concerns (even while reducing others). To explore this question, we interviewed key stakeholders with the aim of identifying potential ethical challenges in the design and conduct of pragmatic RCTs with a view to developing future guidance on these issues. METHODS: Interviews were conducted with clinical investigators, methodologists, patient partners, ethicists, and other knowledge users (e.g., regulators). Interviews covered experiences with pragmatic RCTs, ethical issues relevant to pragmatic RCTs, and perspectives on the appropriate oversight of pragmatic RCTs. Interviews were coded inductively by two coders. Interim and final analyses were presented to the broader team for comment and discussion before the analytic framework was finalized. RESULTS: We conducted 45 interviews between April and September 2018. Interviewees represented a range of disciplines and jurisdictions as well as varying content expertise. Issues of importance in pragmatic RCTs were (1) identification of relevant risks from trial participation and determination of what constitutes minimal risk; (2) determining when alterations to traditional informed consent approaches are appropriate; (3) the distinction between research, quality improvement, and practice; (4) the potential for broader populations to be affected by the trial and what protections they might be owed; (5) the broader range of trial stakeholders in pragmatic RCTs, and determining their roles and responsibilities; and (6) determining what constitutes "usual care" and implications for trial reporting. CONCLUSIONS: Our findings suggest both the need to discuss familiar ethical topics in new ways and that there are new ethical issues in pragmatic RCTs that need greater attention. Addressing the highlighted issues and developing guidance will require multidisciplinary input, including patient and community members, within a broader and more comprehensive analysis that extends beyond consent and attends to the identified considerations relating to risk and stakeholder roles and responsibilities.

Public Approval of Exception From Informed Consent in Emergency Clinical Trials: A Systematic Review of Community Consultation Surveys.

Importance: The US Food and Drug Administration (FDA) created the exception from informed consent (EFIC) pathway in 1996 to allow some emergency trials to enroll patients without informed consent. To protect individual autonomy and preserve public trust, the FDA requires that EFIC trial investigators consult with community members before a trial may begin. Objectives: To analyze data from surveys conducted as part of community consultation ahead of EFIC trials and assess levels of public approval. Data Sources: All trials granted an EFIC must submit documentation of compliance with EFIC regulations to a publicly available docket at the FDA. Submissions between November 1, 1996, and October 23, 2017, were reviewed. Study Selection: Trials with survey data were included. Data Extraction and Synthesis: Data were extracted between January 2018 and June 2018 and were analyzed between June 2018 and August 2018. The quality and validity of data were assessed according to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines. A random-effects metaregression was used to assess the association of demographic characteristics with EFIC approval. Main Outcomes and Measures: The primary study outcome was EFIC approval. Results: The FDA docket contained 15 958 pages of material with survey data for 42 448 individuals submitted by 27 trials. Public approval of EFIC varied by question type, with more people willing to approve initiation of EFIC trials in their community (86.5%) than personal enrollment (73.0%), enrollment of a family member (68.6%), or the principle of enrollment without consent (58.4%) (P < .001 for all comparisons). In the United States, African American individuals made up 29.3% of those enrolled in EFIC trials that reported data on race (5064 of 17 302) but only 16.7% of those surveyed as part of community consultation. In the United States and Canada, men made up 42.9% of the surveyed population but 65.6% of those eventually enrolled in EFIC trials (29 961 of 45 694). Groups surveyed with higher proportions of African American and male respondents had lower rates of EFIC approval. Conclusions and Relevance: Public approval of EFIC trials varied by question type and by the respondents' reported race and sex. The demographic characteristics of those surveyed did not match the demographic characteristics of EFIC enrollees. The FDA could strengthen community consultation by standardizing survey instruments and reporting, requiring broader inclusion of African American and male respondents, clarifying the function of surveys in the development and modification of trial protocols, and building more public consensus around the acceptable use of EFIC.

Assessment of the Role of Niacin in Managing Cardiovascular Disease Outcomes: A Systematic Review and Meta-analysis.

Importance: Niacin remains a therapeutic option for patients with cardiovascular disease, but recent studies have called into question the effectiveness of other drugs that increase high-density lipoprotein cholesterol levels. Objective: To systematically review and evaluate the evidence supporting current US Food and Drug Administration-approved uses of niacin in cardiovascular disease prevention settings. Data Sources: MEDLINE, Embase, Cochrane Controlled Clinical Trial Register (Central), ClinicalTrials.gov, and TrialResults-center, from database inception to October 2017. Study Selection: The systematic review included clinical trials involving niacin as a treatment for cardiovascular disease. The meta-analysis included randomized clinical trials reporting niacin's effect, as exposure, on at least 1 long-term cardiovascular disease outcome. Data Extraction and Synthesis: Aggregate study-level data were extracted between November 2017 and January 2018 by 3 independent reviewers, and the analysis was performed in February 2018. Inverse-variance weighted methods were used to produce pooled risk ratios using random-effects models for between-study heterogeneity. Random effects-weighted metaregression analysis was used to assess the association of change in high-density lipoprotein cholesterol levels with the log risk ratio of the pooled results. Main Outcomes and Measures: Cardiovascular disease, coronary heart disease mortality, and other cardiovascular events, including acute coronary syndrome, fatal and nonfatal stroke, revascularization, and major adverse cardiac events. Results: Of 119 clinical trials, 17 documented niacin's effect on at least 1 cardiovascular disease outcome. The meta-analysis included 35 760 patients with histories of cardiovascular disease or dyslipidemia. Cumulative evidence found no preventive association of niacin with cardiovascular outcomes in secondary prevention. Stratified meta-analysis showed an association of niacin monotherapy with reduction of some cardiovascular events among patients without statin treatment (acute coronary syndrome: relative risk, 0.74; 95% CI, 0.58-0.96; stroke: relative risk, 0.74; 95% CI, 0.59-0.94; revascularization: relative risk, 0.51; 95% CI, 0.37-0.72). These results were mainly derived from 2 trials conducted in the 1970s and 1980s. Conclusions and Relevance: Niacin may have some use in lipid control for secondary prevention as monotherapy, perhaps in patients intolerant to statins, but evidence is from older studies on a population potentially not representative of current-day patients.

A systematic review of trial-level meta-analyses measuring the strength of association between surrogate end-points and overall survival in oncology.

BACKGROUND: Surrogates are frequently used in cancer medicine as the end-point of clinical trials and as the basis of United States Food and Drug Administration approvals, but they do not always represent outcomes that are important for patients. We aim to build upon previous umbrella reviews of surrogate validation studies by identifying and examining all meta-analyses of randomised controlled trials that evaluate the strength of correlation between overall survival (OS) and surrogate markers. METHODS: Google Scholar and PubMed were searched by two independent reviewers for all eligible meta-analyses of randomised controlled trials examining the correlation between a surrogate end-point and OS in medical oncology. Included studies were trial-level (level-1) meta-analyses of randomised controlled trials in cancer. Data abstracted include date of publication, tumour type, setting, trial set, number of studies included in the analysis, dates of included publications, correlation coefficients and method to determine the correlation coefficient. RESULTS: Seventy-eight articles met the inclusion criteria and reported correlations in 89 settings. Eleven (12%) of these validation studies found only high correlation(s), while nine (10%) settings showed a moderate-only correlation. Thirty-four (38%) reported only low correlation(s). Thirty-five (39%) reported correlations of different strengths, depending on surrogate marker used and test of correlation. CONCLUSIONS: In this large, umbrella analysis of surrogate validation studies, we found most surrogates in oncology had low or modest correlation with OS, which suggests that caution should be used when making conclusions based on surrogate markers.