Emulating Control Arms for Cancer Clinical Trials Using External Cohorts Created From Electronic Health Record-Derived Real-World Data.

Electronic health record (EHR)-derived real-world data (RWD) can be sourced to create external comparator cohorts to oncology clinical trials. This exploratory study assessed whether EHR-derived patient cohorts could emulate select clinical trial control arms across multiple tumor types. The impact of analytic decisions on emulation results was also evaluated. By digitizing Kaplan-Meier curves, we reconstructed published control arm results from 15 trials that supported drug approvals from January 1, 2016, to April 30, 2018. RWD cohorts were constructed using a nationwide EHR-derived de-identified database by aligning eligibility criteria and weighting to trial baseline characteristics. Trial data and RWD cohorts were compared using Kaplan-Meier and Cox proportional hazards regression models for progression-free survival (PFS) and overall survival (OS; individual cohorts) and multitumor random effects models of hazard ratios (HRs) for median endpoint correlations (across cohorts). Post hoc, the impact of specific analytic decisions on endpoints was assessed using a case study. Comparing trial data and weighted RWD cohorts, PFS results were more similar (HR range = 0.63-1.18, pooled HR = 0.84, correlation of median = 0.91) compared to OS (HR range = 0.36-1.09, pooled HR = 0.76, correlation of median = 0.85). OS HRs were more variable and trended toward worse for RWD cohorts. The post hoc case study had OS HR ranging from 0.67 (95% confidence interval (CI): 0.56-0.79) to 0.92 (95% CI: 0.78-1.09) depending on specific analytic decisions. EHR-derived RWD can emulate oncology clinical trial control arm results, although with variability. Visibility into clinical trial cohort characteristics may shape and refine analytic approaches.

Characterization of a Real-World Response Variable and Comparison with RECIST-Based Response Rates from Clinical Trials in Advanced NSCLC.

INTRODUCTION: Effectiveness metrics for real-word research, analogous to clinical trial ones, are needed. This study aimed to develop a real-world response (rwR) variable applicable to solid tumors and to evaluate its clinical relevance and meaningfulness. METHODS: This retrospective study used patient cohorts with advanced non-small cell lung cancer from a nationwide, de-identified electronic health record (EHR)-derived database. Disease burden information abstracted manually was classified into response categories anchored to discrete therapy lines (per patient-line). In part 1, we quantified the feasibility and reliability of data capture, and estimated the association between rwR status and real-world progression-free survival (rwPFS) and real-world overall survival (rwOS). In part 2, we investigated the correlation between published clinical trial overall response rates (ORRs) and real-world response rates (rwRRs) from corresponding real-world patient cohorts. RESULTS: In part 1, 85.4% of patients (N = 3248) had at least one radiographic assessment documented. Median abstraction time per patient-line was 15.0 min (IQR 7.8-28.1). Inter-abstractor agreement on presence/absence of at least one assessment was 0.94 (95% CI 0.92-0.96; n = 503 patient-lines abstracted in duplicate); inter-abstractor agreement on best confirmed response category was 0.82 (95% CI 0.78-0.86; n = 384 with at least one captured assessment). Confirmed responders at a 3-month landmark showed significantly lower risk of death and progression in rwOS and rwPFS analyses across all line settings. In part 2, rwRRs (from 12 rw cohorts) showed a high correlation with trial ORRs (Spearman's ρ = 0.99). CONCLUSIONS: We developed a rwR variable generated from clinician assessments documented in EHRs following radiographic evaluations. This variable provides clinically meaningful information and may provide a real-world measure of treatment effectiveness.

Validation of diagnosis codes to identify side of colon in an electronic health record registry.

BACKGROUND: The use of real-world data to generate evidence requires careful assessment and validation of critical variables before drawing clinical conclusions. Prospective clinical trial data suggest that anatomic origin of colon cancer impacts prognosis and treatment effectiveness. As an initial step in validating this observation in routine clinical settings, we explored the feasibility and accuracy of obtaining information on tumor sidedness from electronic health records (EHR) billing codes. METHODS: Nine thousand four hundred three patients with metastatic colorectal cancer (mCRC) were selected from the Flatiron Health database, which is derived from de-identified EHR data. This study included a random sample of 200 mCRC patients. Tumor site data derived from International Classification of Diseases (ICD) codes were compared with data abstracted from unstructured documents in the EHR (e.g. surgical and pathology notes). Concordance was determined via observed agreement and Cohen's kappa coefficient (κ). Accuracy of ICD codes for each tumor site (left, right, transverse) was determined by calculating the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), and corresponding 95% confidence intervals, using abstracted data as the gold standard. RESULTS: Study patients had similar characteristics and side of colon distribution compared with the full mCRC dataset. The observed agreement between the ICD codes and abstracted data for tumor site for all sampled patients was 0.58 (κ = 0.41). When restricting to the 62% of patients with a side-specific ICD code, the observed agreement was 0.84 (κ = 0.79). The specificity (92-98%) of structured data for tumor location was high, with lower sensitivity (49-63%), PPV (64-92%) and NPV (72-97%). Demographic and clinical characteristics were similar between patients with specific and non-specific side of colon ICD codes. CONCLUSIONS: ICD codes are a highly reliable indicator of tumor location when the specific location code is entered in the EHR. However, non-specific side of colon ICD codes are present for a sizable minority of patients, and structured data alone may not be adequate to support testing of some research hypotheses. Careful assessment of key variables is required before determining the need for clinical abstraction to supplement structured data in generating real-world evidence from EHRs.

Generating Real-World Tumor Burden Endpoints from Electronic Health Record Data: Comparison of RECIST, Radiology-Anchored, and Clinician-Anchored Approaches for Abstracting Real-World Progression in Non-Small Cell Lung Cancer.

INTRODUCTION: Real-world evidence derived from electronic health records (EHRs) is increasingly recognized as a supplement to evidence generated from traditional clinical trials. In oncology, tumor-based Response Evaluation Criteria in Solid Tumors (RECIST) endpoints are standard clinical trial metrics. The best approach for collecting similar endpoints from EHRs remains unknown. We evaluated the feasibility of a RECIST-based methodology to assess EHR-derived real-world progression (rwP) and explored non-RECIST-based approaches. METHODS: In this retrospective study, cohorts were randomly selected from Flatiron Health's database of de-identified patient-level EHR data in advanced non-small cell lung cancer. A RECIST-based approach tested for feasibility (N = 26). Three non-RECIST approaches were tested for feasibility, reliability, and validity (N = 200): (1) radiology-anchored, (2) clinician-anchored, and (3) combined. Qualitative and quantitative methods were used. RESULTS: A RECIST-based approach was not feasible: cancer progression could be ascertained for 23% (6/26 patients). Radiology- and clinician-anchored approaches identified at least one rwP event for 87% (173/200 patients). rwP dates matched 90% of the time. In 72% of patients (124/173), the first clinician-anchored rwP event was accompanied by a downstream event (e.g., treatment change); the association was slightly lower for the radiology-anchored approach (67%; 121/180). Median overall survival (OS) was 17 months [95% confidence interval (CI) 14, 19]. Median real-world progression-free survival (rwPFS) was 5.5 months (95% CI 4.6, 6.3) and 4.9 months (95% CI 4.2, 5.6) for clinician-anchored and radiology-anchored approaches, respectively. Correlations between rwPFS and OS were similar across approaches (Spearman's rho 0.65-0.66). Abstractors preferred the clinician-anchored approach as it provided more comprehensive context. CONCLUSIONS: RECIST cannot adequately assess cancer progression in EHR-derived data because of missing data and lack of clarity in radiology reports. We found a clinician-anchored approach supported by radiology report data to be the optimal, and most practical, method for characterizing tumor-based endpoints from EHR-sourced data. FUNDING: Flatiron Health Inc., which is an independent subsidiary of the Roche group.

An evaluation of the impact of missing deaths on overall survival analyses of advanced non-small cell lung cancer patients conducted in an electronic health records database.

PURPOSE: The aim of this study was to assess the impact of missing death data on survival analyses conducted in an oncology EHR-derived database. METHODS: The study was conducted using the Flatiron Health oncology database and the National Death Index (NDI) as a gold standard. Three analytic frameworks were evaluated in advanced non-small cell lung cancer (aNSCLC) patients: median overall survival [mOS]), relative risk estimates conducted within the EHR-derived database, and "external control arm" analyses comparing an experimental group augmented with mortality data from the gold standard to a control group from the EHR-derived database only. The hazard ratios (HRs) obtained within the EHR-derived database (91% sensitivity) and the external control arm analyses, were compared with results when both groups were augmented with mortality data from the gold standard. The above analyses were repeated using simulated lower mortality sensitivities to understand the impact of more extreme levels of missing deaths. RESULTS: Bias in mOS ranged from modest (0.6-0.9 mos.) in the EHR-derived cohort with (91% sensitivity) to substantial when lower sensitivities were generated through simulation (3.3-9.7 mos.). Overall, small differences were observed in the HRs for the EHR-derived cohort across comparative analyses when compared with HRs obtained using the gold standard data source. When only one treatment arm was subject to estimation bias, the bias was slightly more pronounced, but increased substantially when lower sensitivities were simulated. CONCLUSIONS: The impact on survival analysis is minimal with high mortality sensitivity with only modest impact associated within external control arm applications.

Real-World Outcomes of Patients with Metastatic Non-Small Cell Lung Cancer Treated with Programmed Cell Death Protein 1 Inhibitors in the Year Following U.S. Regulatory Approval.

BACKGROUND: Evidence from cancer clinical trials has strong internal validity but can be difficult to generalize to real-world patient populations. Here we analyzed real-world outcomes of patients with metastatic non-small cell lung cancer (mNSCLC) treated with programmed cell death protein 1 (PD-1) inhibitors in the first year following U.S. regulatory approval. MATERIALS AND METHODS: This retrospective study leveraged electronic health record (EHR) data collected during routine patient care in community cancer care clinics. The cohort included patients with mNSCLC who had received nivolumab or pembrolizumab for metastatic disease (n = 1,344) with >1 EHR-documented visit from January 1, 2011, to March 31, 2016. Patients with a > 90-day gap between advanced disease diagnosis and first EHR structured data entry were excluded. RESULTS: Estimated median overall survival (OS) was 8.0 months (95% confidence interval 7.4-9.0 months). Estimated median OS was 4.7 months (3.4-6.6) for patients with anaplastic lymphoma kinase rearrangement- and epidermal growth factor receptor mutation-positive tumors, and 8.6 months (7.7-10.6) for patients without such mutations. Age at PD-1 inhibitor initiation or line of therapy did not impact OS. CONCLUSION: This analysis suggests OS in real-world patients may be shorter than in conventional clinical trial patient cohorts, potentially due to narrow trial eligibility criteria. The lack of difference in OS by line of therapy or age at immunotherapy initiation suggests sustained benefit of PD-1 inhibitors in multitreated patients with mNSCLC and that age is not a predictor of outcome. Further studies are underway in patients with comorbidities, organ dysfunction, and multiple prior therapies. IMPLICATIONS FOR PRACTICE: This study evaluated data derived from electronic health records of patients with metastatic non-small cell lung cancer treated with programmed cell death protein 1 (PD-1) inhibitors in the year following regulatory approval. This real-world cohort had shorter overall survival (OS) indexed to PD-1 inhibitor initiation than reported in clinical trials. Late-line treatment did not influence OS, and patients aged >75 at immunotherapy initiation did not have worse outcomes than younger patients. As new therapies enter clinical practice, real-world data can complement clinical trial evidence providing information on generalizability and helping inform clinical treatment decisions.

Development and Validation of a High-Quality Composite Real-World Mortality Endpoint.

OBJECTIVE: To create a high-quality electronic health record (EHR)-derived mortality dataset for retrospective and prospective real-world evidence generation. DATA SOURCES/STUDY SETTING: Oncology EHR data, supplemented with external commercial and US Social Security Death Index data, benchmarked to the National Death Index (NDI). STUDY DESIGN: We developed a recent, linkable, high-quality mortality variable amalgamated from multiple data sources to supplement EHR data, benchmarked against the highest completeness U.S. mortality data, the NDI. Data quality of the mortality variable version 2.0 is reported here. PRINCIPAL FINDINGS: For advanced non-small-cell lung cancer, sensitivity of mortality information improved from 66 percent in EHR structured data to 91 percent in the composite dataset, with high date agreement compared to the NDI. For advanced melanoma, metastatic colorectal cancer, and metastatic breast cancer, sensitivity of the final variable was 85 to 88 percent. Kaplan-Meier survival analyses showed that improving mortality data completeness minimized overestimation of survival relative to NDI-based estimates. CONCLUSIONS: For EHR-derived data to yield reliable real-world evidence, it needs to be of known and sufficiently high quality. Considering the impact of mortality data completeness on survival endpoints, we highlight the importance of data quality assessment and advocate benchmarking to the NDI.

Characteristics of Real-World Metastatic Non-Small Cell Lung Cancer Patients Treated with Nivolumab and Pembrolizumab During the Year Following Approval.

BACKGROUND: Evidence from cancer clinical trials can be difficult to generalize to real-world patient populations, but can be complemented by real-world evidence to optimize personalization of care. Further, real-world usage patterns of programmed cell death protein 1 (PD-1) inhibitors following approval can inform future studies of subpopulations underrepresented in clinical trials. MATERIALS AND METHODS: We performed a multicenter analysis using electronic health record data collected during routine care of patients treated in community cancer care clinics in the Flatiron Health network. Real-world metastatic non-small cell lung cancer (NSCLC) patients who received nivolumab or pembrolizumab in the metastatic setting (n = 1,344) were selected from a starting random sample of 55,969 NSCLC patients with two or more documented visits from January 1, 2011, through March 31, 2016. The primary study outcome measurement was demographic and treatment characteristics of the cohort. RESULTS: Median age at PD-1 inhibitor initiation was 69 years (interquartile range 61-75). Patients were 56% male, 88% smokers, 65% nonsquamous histology, and 64% diagnosed at stage IV. Of 1,344 patients, 112 (8%) were tested for programmed death-ligand 1 expression. Overall, 50% received nivolumab or pembrolizumab in the second line, with a substantial proportion of third and later line use that began to decline in Q4 2015. CONCLUSION: During the year following U.S. regulatory approval of PD-1 inhibitors for treatment of NSCLC, real-world patients receiving nivolumab or pembrolizumab were older at treatment initiation and more had smoking history relative to clinical trial cohorts. Studies of outcomes in underrepresented subgroups are needed to inform real-world treatment decisions. IMPLICATIONS FOR PRACTICE: Evidence gathered in conventional clinical trials used to assess safety and efficacy of new therapies is not necessarily generalizable to real-world patients receiving these drugs following regulatory approval. Real-world evidence derived from electronic health record data can yield complementary evidence to enable optimal clinical decisions. Examined here is a cohort of programmed cell death protein 1 inhibitor-treated metastatic non-small cell lung cancer patients in the first year following regulatory approval of these therapies in this indication. The analysis revealed how the real-world cohort differed from the clinical trial cohorts, which will inform which patients are underrepresented and warrant additional studies.