Visualizing External Validity: Graphical Displays to Inform the Extension of Treatment Effects from Trials to Clinical Practice.

BACKGROUND: In the presence of effect measure modification, estimates of treatment effects from randomized controlled trials may not be valid in clinical practice settings. The development and application of quantitative approaches for extending treatment effects from trials to clinical practice settings is an active area of research. METHODS: In this article, we provide researchers with a practical roadmap and four visualizations to assist in variable selection for models to extend treatment effects observed in trials to clinical practice settings and to assess model specification and performance. We apply this roadmap and visualizations to an example extending the effects of adjuvant chemotherapy (5-fluorouracil vs. plus oxaliplatin) for colon cancer from a trial population to a population of individuals treated in community oncology practices in the United States. RESULTS: The first visualization screens for potential effect measure modifiers to include in models extending trial treatment effects to clinical practice populations. The second visualization displays a measure of covariate overlap between the clinical practice populations and the trial population. The third and fourth visualizations highlight considerations for model specification and influential observations. The conceptual roadmap describes how the output from the visualizations helps interrogate the assumptions required to extend treatment effects from trials to target populations. CONCLUSIONS: The roadmap and visualizations can inform practical decisions required for quantitatively extending treatment effects from trials to clinical practice settings.

Effectiveness of adjuvant FOLFOX vs 5FU/LV in adults over age 65 with stage II and III colon cancer using a novel hybrid approach.

PURPOSE: Estimates of cancer therapy effects can differ in clinical trials and clinical practice, partly due to underrepresentation of certain patient subgroups in trials. We utilize a hybrid approach, combining clinical trial and real-world data, to estimate the comparative effectiveness of two adjuvant chemotherapy regimens for colon cancer. METHODS: We identified patients aged 66 and older enrolled in the Multicenter International Study of Oxaliplatin/5FU-LV in the Adjuvant Treatment of Colon Cancer. Similar patients were identified in the Surveillance, Epidemiology, and End Results (SEER)-Medicare database, initiating adjuvant chemotherapy with either 5-fluorouracil (5FU) alone or in combination with oxaliplatin (FOLFOX). We used logistic regression to estimate the likelihood of trial enrollment as a function of age, sex, and substage. Using inverse odds of sampling weights (IOSW), we compared 5-year mortality in patients randomized to FOLFOX vs 5FU using weighted Cox proportional hazards regression, the Nelson-Aalen estimator for cumulative hazards, and bootstrapping for 95% confidence intervals (CIs). RESULTS: There were 690 trial participants and 3834 SEER-Medicare patients. The SEER-Medicare population was older and had a higher proportion of stage IIIB and IIIC patients than the trial. After controlling for differences between populations, the IOSW 5-year HR was 1.21 (0.89, 1.65), slightly farther from the null than the trial estimate (HR = 1.14, 95%CI: 0.87, 1.49). CONCLUSIONS: This study supports mounting evidence of little to no incremental reduction in 5-year mortality for FOLFOX vs 5FU in older adults with stage II-III colon cancer, emphasizing the importance of combining clinical trial and real-world data to support such conclusions.

Diagnostic Assessment of Assumptions for External Validity: An Example Using Data in Metastatic Colorectal Cancer.

BACKGROUND: Methods developed to estimate intervention effects in external target populations assume that all important effect measure modifiers have been identified and appropriately modeled. Propensity score-based diagnostics can be used to assess the plausibility of these assumptions for weighting methods. METHODS: We demonstrate the use of these diagnostics when assessing the transportability of treatment effects from the standard of care for metastatic colorectal cancer control arm in a phase III trial (HORIZON III) to a target population of 1,942 Medicare beneficiaries age 65+ years. RESULTS: In an unadjusted comparison, control arm participants had lower mortality compared with target population patients treated with the standard of care therapy (trial vs. target hazard ratio [HR] = 0.72, 95% confidence interval [CI], 0.58, 0.89). Applying inverse odds of sampling weights attenuated the trial versus target HR (weighted HR = 0.96, 95% CI = 0.73, 1.26). However, whether unadjusted or weighted, hazards did not appear proportional. At 6 months of follow-up, mortality was lower in the weighted trial population than the target population (weighted trial vs. target risk difference [RD] = -0.07, 95% CI = -0.13, -0.01), but not at 12 months (weighted RD = 0.00, 95% CI = -0.09, 0.09). CONCLUSION: These diagnostics suggest that direct transport of treatment effects from HORIZON III to the Medicare population is not valid. However, the proposed sampling model might allow valid transport of the treatment effects on longer-term mortality from HORIZON III to the Medicare population treated in clinical practice. See video abstract at, http://links.lww.com/EDE/B435.