Assessing the performance of physician's prescribing preference as an instrumental variable in comparative effectiveness research with moderate and small sample sizes: a simulation study.

Aim: This simulation study is to assess the utility of physician's prescribing preference (PPP) as an instrumental variable for moderate and smaller sample sizes. Materials & methods: We designed a simulation study to imitate a comparative effectiveness research under different sample sizes. We compare the performance of instrumental variable (IV) and non-IV approaches using two-stage least squares (2SLS) and ordinary least squares (OLS) methods, respectively. Further, we test the performance of different forms of proxies for PPP as an IV. Results: The percent bias of 2SLS is around approximately 20%, while the percent bias of OLS is close to 60%. The sample size is not associated with the level of bias for the PPP IV approach. Conclusion: Irrespective of sample size, the PPP IV approach leads to less biased estimates of treatment effectiveness than OLS adjusting for known confounding only. Particularly for smaller sample sizes, we recommend constructing PPP from long prescribing histories to improve statistical power.

Augmenting external control arms using Bayesian borrowing: a case study in first-line non-small cell lung cancer.

Aim: This study aimed to improve comparative effectiveness estimates and discuss challenges encountered through the application of Bayesian borrowing (BB) methods to augment an external control arm (ECA) constructed from real-world data (RWD) using historical clinical trial data in first-line non-small-cell lung cancer (NSCLC). Materials & methods: An ECA for a randomized controlled trial (RCT) in first-line NSCLC was constructed using ConcertAI Patient360™ to assess chemotherapy with or without cetuximab, in the bevacizumab-inappropriate subpopulation. Cardinality matching was used to match patient characteristics between the treatment arm (cetuximab + chemotherapy) and ECA. Overall survival (OS) was assessed as the primary outcome using Cox proportional hazards (PH). BB was conducted using a static power prior under a Weibull PH parameterization with borrowing weights from 0.0 to 1.0 and augmentation of the ECA from a historical control trial. Results: The constructed ECA yielded a higher overall survival (OS) hazard ratio (HR) (HR = 1.53; 95% CI: 1.21-1.93) than observed in the matched population of the RCT (HR = 0.91; 95% CI: 0.73-1.13). The OS HR decreased through the incorporation of BB (HR = 1.30; 95% CI: 1.08-1.54, borrowing weight = 1.0). BB was applied to augment the RCT control arm via a historical control which improved the precision of the observed HR estimate (1.03; 95% CI: 0.86-1.22, borrowing weight = 1.0), in comparison to the matched population of the RCT alone. Conclusion: In this study, the RWD ECA was unable to successfully replicate the OS estimates from the matched population of the selected RCT. The inability to replicate could be due to unmeasured confounding and variations in time-periods, follow-up and subsequent therapy. Despite these findings, we demonstrate how BB can improve precision of comparative effectiveness estimates, potentially aid as a bias assessment tool and mitigate challenges of traditional methods when appropriate external data sources are available.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 15.

In this latest update we discuss real-world evidence (RWE) guidance from the leading oncology professional societies, the American Society of Clinical Oncology and the European Society for Medical Oncology, and the PRINCIPLED practical guide on the design and analysis of causal RWE studies.

[Designing observational studies by target trial emulation]. / De hypothetische trial.

Ideally, causal research questions will be answered with randomized trials, but this is not always feasible for practical, ethical, or methodological reasons. To obtain a reliable answer to causal questions with observational data, target trial emulation has been introduced, in which an observational study is designed, conducted, and analyzed emulating the target trial. After phrasing a causal question, this framework first addresses seven components of the target trial protocol: eligibility criteria, treatment strategies, assignment procedures, follow-up period, outcome of interest, causal contrast of interest, and statistical analysis plan. Subsequently, these elements are emulated in the observational study. This approach addresses methodological pitfalls before initiating the study, draws more unambiguous conclusions, and provides a structured assessment of limitations of observational studies as well as randomized trials. The use of the target trial framework to support the design of observational studies is increasing.

Patient participation impacts outcome domain selection in core outcome sets for research: an updated systematic review.

OBJECTIVES: The importance of including patients, carers, and the public in health research is well recognized, including the need to consider outcomes in health care research that reflect the priorities of patients. Core outcome sets (COS) define the minimum set of outcomes that should be measured and reported in research of a given condition, determined through consensus among key stakeholders. The Core Outcome Measures in Effectiveness Trials Initiative undertakes an annual systematic review (SR) to identify newly published COS to update its online database of COS for research. The objective of this study was to assess the impact of patient participation on COS. STUDY DESIGN AND SETTING: SR methods used in previous updates were applied to identify research studies published or indexed in 2020 and 2021 (conducted as separate reviews) that report development of a COS, regardless of any specifications relating to condition, population, intervention, or setting. Studies were assessed according to published standards for COS development, and core outcomes extracted from study publications were categorized according to an outcome taxonomy and added to an existing database of core outcome classifications of all previously published COS. The effect of patient participation on core domains was examined. RESULTS: Searches identified 56 new studies published in 2020 and 54 in 2021. All studies met all four minimum standards for scope, and 42 (75%) of the 2020 studies and 45 (83%) of the 2021 studies met all three standards for stakeholders involved. However, only 19 (34%) of the 2020 studies and 18 (33%) of the 2021 studies met all four standards for the consensus process. COS that involved patients or their representatives are more likely to include life impact outcomes (239, 86%) than COS without patient participation (193, 62%). Physiological/clinical outcomes are almost always specified at a granular level, whereas life impact outcomes are often described at a higher level. CONCLUSION: This study adds to the body of evidence demonstrating the importance and impact of including patients, carers, and the public in COS development, in particular by demonstrating that the impact of interventions on patients' lives is more likely to be represented in COS that involve patients or their representatives. COS developers are encouraged to pay increased attention to methods and reporting relating to the consensus process. Further work is required to understand the appropriateness and rationale for the discrepancy in granularity levels between outcome domains.

An evaluation of stakeholder engagement in comparative effectiveness research: lessons learned from SWOG S1415CD.

Aim: Stakeholder engagement is central to comparative effectiveness research yet there are gaps in definitions of success. We used a framework developed by Lavallee et al. defining effective engagement criteria to evaluate stakeholder engagement during a pragmatic cluster-randomized trial. Methods: Semi-structured interviews were developed from the framework and completed to learn about members' experiences. Interviews were analyzed in a deductive approach for themes related to the effective engagement criteria. Results: Thirteen members participated and described: respect for ideas, time to achieve consensus, access to information and continuous feedback as areas of effective engagement. The primary criticism was lack of diversity. Discussion: Feedback was positive, particularly among themes of respect, trust and competence, and led to development of a list of best practices for engagement. The framework was successful for evaluating engagement. Conclusion: Standardized frameworks allow studies to formally evaluate their stakeholder engagement approach and develop best practices for future research.

What is this article about? This article is about the evaluation of how effective the stakeholder engagement was in a comparative effectiveness research (CER) study funded by the Patient Centered Outcomes Research Institute (PCORI). The research team found a framework (developed by Lavalle et al.) that defined six different criteria for effective stakeholder engagement, and used that criteria to complete semi-structured interviews with the stakeholders involved with our study. These interviews were reviewed to determine what stakeholder engagement processes were successful and helped provide a list of best practices for stakeholder engagement for other researchers doing CER. What were the results? Stakeholders highlighted respect for their ideas, time to achieve consensus, easy access to information and a continuous feedback loop between study team and stakeholders as effective engagement processes. What do the results mean? These results can help other researchers doing CER learn best practices to implement from the outset of a study to best engage stakeholders in their research. The results also show that having a standardized framework to evaluate stakeholder engagement is important and allows for research teams to formally evaluate their engagement approach and learn what was successful and where there are areas for improvement in future studies.

ISPE-Endorsed Guidance in Using Electronic Health Records for Comparative Effectiveness Research in COVID-19: Opportunities and Trade-Offs.

As the scientific research community along with healthcare professionals and decision makers around the world fight tirelessly against the coronavirus disease 2019 (COVID-19) pandemic, the need for comparative effectiveness research (CER) on preventive and therapeutic interventions for COVID-19 is immense. Randomized controlled trials markedly under-represent the frail and complex patients seen in routine care, and they do not typically have data on long-term treatment effects. The increasing availability of electronic health records (EHRs) for clinical research offers the opportunity to generate timely real-world evidence reflective of routine care for optimal management of COVID-19. However, there are many potential threats to the validity of CER based on EHR data that are not originally generated for research purposes. To ensure unbiased and robust results, we need high-quality healthcare databases, rigorous study designs, and proper implementation of appropriate statistical methods. We aimed to describe opportunities and challenges in EHR-based CER for COVID-19-related questions and to introduce best practices in pharmacoepidemiology to minimize potential biases. We structured our discussion into the following topics: (1) study population identification based on exposure status; (2) ascertainment of outcomes; (3) common biases and potential solutions; and (iv) data operational challenges specific to COVID-19 CER using EHRs. We provide structured guidance for the proper conduct and appraisal of drug and vaccine effectiveness and safety research using EHR data for the pandemic. This paper is endorsed by the International Society for Pharmacoepidemiology (ISPE).

Propensity score methods for comparative-effectiveness analysis: A case study of direct oral anticoagulants in the atrial fibrillation population.

OBJECTIVE: To explore methodological challenges when using real-world evidence (RWE) to estimate comparative-effectiveness in the context of Health Technology Assessment of direct oral anticoagulants (DOACs) in Scotland. METHODS: We used linkage data from the Prescribing Information System (PIS), Scottish Morbidity Records (SMR) and mortality records for newly anticoagulated patients to explore methodological challenges in the use of Propensity score (PS) matching, Inverse Probability Weighting (IPW) and covariate adjustment with PS. Model performance was assessed by standardised difference. Clinical outcomes (stroke and major bleeding) and mortality were compared for all DOACs (including apixaban, dabigatran and rivaroxaban) versus warfarin. Patients were followed for 2 years from first oral anticoagulant prescription to first clinical event or death. Censoring was applied for treatment switching or discontinuation. RESULTS: Overall, a good balance of patients' covariates was obtained with every PS model tested. IPW was found to be the best performing method in assessing covariate balance when applied to subgroups with relatively large sample sizes (combined-DOACs versus warfarin). With the IPTW-IPCW approach, the treatment effect tends to be larger, but still in line with the treatment effect estimated using other PS methods. Covariate adjustment with PS in the outcome model performed well when applied to subgroups with smaller sample sizes (dabigatran versus warfarin), as this method does not require further reduction of sample size, and trimming or truncation of extreme weights. CONCLUSION: The choice of adequate PS methods may vary according to the characteristics of the data. If assumptions of unobserved confounding hold, multiple approaches should be identified and tested. PS based methods can be implemented using routinely collected linked data, thus supporting Health Technology decision-making.

The Use of Active Comparators in Self-Controlled Designs.

For self-controlled studies of medication-related effects, time-varying confounding by indication can occur if the indication varies over time. We describe how active comparators might mitigate such bias, using an empirical example. Approaches to using active comparators are described for case-crossover design, case-time-control design, self-controlled case-series, and sequence symmetry analyses. In the empirical example, we used Danish data from 1996-2018 to study the association between penicillin and venous thromboembolism (VTE), using roxithromycin, a macrolide antibiotic, as comparator. Upper respiratory infection is a transient risk factor for VTE, thus representing time-dependent confounding by indication. Odds ratios for case-crossover analysis were 3.35 (95% confidence interval: 3.23, 3.49) for penicillin and 3.56 (95% confidence interval: 3.30, 3.83) for roxithromycin. We used a Wald-based method or an interaction term to estimate the odds ratio for penicillin with roxithromycin as comparator. These 2 estimates were 0.94 (95% confidence interval: 0.87, 1.03) and 1.03 (95% confidence interval: 0.95, 1.13). Results were similar for the case-time-control analysis, but both the self-controlled case-series and sequence symmetry analysis suggested a weak protective effect of penicillin, seemingly explained by VTE affecting future exposure exclusively for penicillin. The strong association of antibiotics with VTE suggests presence of confounding by indication. Such confounding can be mitigated by using an active comparator.

A novel tool to assess the quality of RWE to guide the management of retinal disease.

Despite the growing importance of real-world evidence (RWE) for guiding clinical decisions in retinal disease, there is currently no widely used guidance available for assessing the quality and relevance of RWE studies in ophthalmology. This paper summarizes the development of a user-friendly tool that facilitates assessment of the quality of available RWE for neovascular age-related macular degeneration (nAMD), diabetic macular oedema (DME) and retinal vein occlusion (RVO). A literature search was conducted to identify tools developed to assess the quality of RWE, in order to identify the most appropriate framework on which to base this tool. The Good Research for Comparative Effectiveness (GRACE) guidelines was chosen for this purpose as it is designed to assess the quality of observational studies and has been extensively validated, including demonstration of strong sensitivity and specificity. The GRACE guidelines were adapted to develop a straightforward tabular tool that allows simple assessment and comparison of the quality of published evidence in retinal disease for researchers and physicians alike, and includes guidance on treatment details, outcome measures, study population, and controlling for bias. The newly developed tool provides a simple method to support assessment of the strength of evidence and certainty of conclusions drawn from RWE in retinal disease, to ensure clinical decision-making is influenced by the highest quality evidence.

Is Real-World Evidence Used in P&T Monographs and Therapeutic Class Reviews?

BACKGROUND: Payers are faced with making coverage and reimbursement decisions based on the best available evidence. Often these decisions apply to patient populations, provider networks, and care settings not typically studied in clinical trials. Treatment effectiveness evidence is increasingly available from electronic health records, registries, and administrative claims. However, little is known about when and what types of real-world evidence (RWE) studies inform pharmacy and therapeutic (P&T) committee decisions. OBJECTIVE: To evaluate evidence sources cited in P&T committee monographs and therapeutic class reviews and assess the design features and quality of cited RWE studies. METHODS: A convenience sample of representatives from pharmacy benefit management, health system, and health plan organizations provided recent P&T monographs and therapeutic class reviews (or references from such documents). Two investigators examined and grouped references into major categories (published studies, unpublished studies, and other/unknown) and multiple subcategories (e.g., product label, clinical trials, RWE, systematic reviews). Cited comparative RWE was reviewed to assess design features (e.g., population, data source, comparators) and quality using the Good ReseArch for Comparative Effectiveness (GRACE) Checklist. RESULTS: Investigators evaluated 565 references cited in 27 monographs/therapeutic class reviews from 6 managed care organizations. Therapeutic class reviews mostly cited published clinical trials (35.3%, 155/439), while single-product monographs relied most on manufacturer-supplied information (42.1%, 53/126). Published RWE comprised 4.8% (21/439) of therapeutic class review references, and none (0/126) of the monograph references. Of the 21 RWE studies, 12 were comparative and assessed patient care settings and outcomes typically not included in clinical trials (community ambulatory settings [10], long-term safety [8]). RWE studies most frequently were based on registry data (6), conducted in the United States (6), and funded by the pharmaceutical industry (5). GRACE Checklist ratings suggested the data and methods of these comparative RWE studies were of high quality. CONCLUSIONS: RWE was infrequently cited in P&T materials, even among therapeutic class reviews where RWE is more readily available. Although few P&T materials cited RWE, the comparative RWE studies were generally high quality. More research is needed to understand when and what types of real-world studies can more routinely inform coverage and reimbursement decisions. DISCLOSURES: This project was funded by the National Pharmaceutical Council. Hurwitz, Brown, Peters, and Malone have nothing to disclose. Graff is employed by the National Pharmaceutical Council Part of this study was presented as a poster presentation at the AMCP Managed Care & Specialty Pharmacy 2016 Annual Meeting; April 19-22, 2016; San Francisco, CA. Study concept and design were primarily contributed by Malone and Graff, along with Hurwitz and Brown. All authors participated in data collection, and data interpretation was performed by Malone, Hurwitz, and Graff, with assistance from Brown and Peters. The manuscript was written primarily by Hurwitz and Malone, along with Graff, Brown, and Peters, and revised by Malone, Brown, Peters, Hurwitz, and Graff.

The use of real-world evidence in ICER's scoping process and clinical evidence assessments.

BACKGROUND: There has been growing interest in using real-world evidence (RWE) for health technology assessment (HTA) in the United States. The Institute for Clinical and Economic Review (ICER) is an independent U.S.-based HTA organization that focuses primarily on pharmaceuticals. RWE is used to inform ICER's scoping and comparative clinical effectiveness (CCE) assessments, but the extent to which it is used has not been quantified. OBJECTIVE: To systematically evaluate use of RWE in the scoping and CCE assessment sections of the ICER HTA reports on pharmaceuticals. METHODS: We reviewed all ICER reports of pharmaceuticals published between January 2014 and June 2019. We examined the average number of instances and the proportion of RWE use in the scoping documents to inform the population, intervention, comparator, outcome, setting, or timing (PICOTS) elements of the appraisal. We also examined the average number of instances and the proportion of RWE use in the CCE assessments to inform effectiveness, safety, or treatment patterns. Finally, we evaluated use of RWE in clinical guidelines that were cited in the CCE assessments. RESULTS: In ICER scoping documents, the mean (SD) number of instances of RWE use was 3.8 (3.7) per document (55% for outcomes, 20% for population, 14% for comparator, 11% for intervention, and 0% for timing and setting). In ICER CCE assessments, the mean (SD) number of instances per assessment was 0.7 (0.5) per drug (53% for effectiveness, 44% for safety, and 3% for treatment patterns). In clinical guidelines used in ICER reports, the mean (SD) number of instances of RWE use was 1.6 (2.3) per drug per guideline (41% for effectiveness, 30% for safety, and 29% for treatment patterns). CONCLUSIONS: RWE was frequently used in the ICER scoping process, particularly to inform selection of outcomes. RWE was used infrequently in ICER CCE assessments, while more often used to inform effectiveness, safety, and treatment patterns in relevant clinical guidelines. There are opportunities to increase the use of RWE in U.S. HTA processes. DISCLOSURES: This study was supported by the Health Tech Fund, University of Washington School of Pharmacy, which was created through unrestricted support from several health care industry companies. Veenstra and Carlson report grant support from the Institute for Clinical and Economic Review outside the submitted work. Carlson reports personal fees from Bayer, Allergan, and Galderma outside the submitted work. Jiao, Lee, and Devine report no support outside the submitted work.

Comparison of Methods for Estimating Therapy Effects by Indirect Comparisons: A Simulation Study.

Objective. In evidence synthesis, therapeutic options have to be compared despite the lack of head-to-head trials. Indirect comparisons are then widely used, although little is known about their performance in situations where cross-trial differences or effect modification are present. Methods. We contrast the matching adjusted indirect comparison (MAIC), simulated treatment comparison (STC), and the method of Bucher using a simulation study. The different methods are evaluated according to their power and type I error rate as well as with respect to the coverage, bias, and the root mean squared error (RMSE) of the effect estimate for practically relevant scenarios using binary and time-to-event endpoints. In addition, we investigate how the power planned for the head-to-head trials influences the actual power of the indirect comparison. Results. Indirect comparisons are considerably underpowered. None of the methods had substantially superior performance. In situations without cross-trial differences and effect modification, MAIC and Bucher led to similar results, while Bucher has the advantage of preserving the within-study randomization. MAIC and STC could enhance power in some scenarios but at the cost of a potential type I error inflation. Adjusting MAIC and STC for confounders that did not modify the effect led to higher bias and RMSE. Conclusion. The choice of effect modifiers in MAIC and STC influences the precision of the indirect comparison. Therefore, a careful selection of effect modifiers is warranted. In addition, missed differences between trials may lead to low power and partly high bias for all considered methods, and thus, results of indirect comparisons should be interpreted with caution.

The RICH LIFE Project: A cluster randomized pragmatic trial comparing the effectiveness of health system only vs. health system Plus a collaborative/stepped care intervention to reduce hypertension disparities.

Disparities in the control of hypertension and other cardiovascular disease risk factors are well-documented in the United States, even among patients seen regularly in the healthcare system. Few existing approaches explicitly address disparities in hypertension care and control. This paper describes the RICH LIFE Project (Reducing Inequities in Care of Hypertension: Lifestyle Improvement for Everyone) design. METHODS: RICH LIFE is a two-arm, cluster-randomized trial, comparing the effectiveness of enhanced standard of care, "Standard of Care Plus" (SCP), to a multi-level intervention, "Collaborative Care/Stepped Care" (CC/SC), for improving blood pressure (BP) control and patient activation and reducing disparities in BP control among 1890 adults with uncontrolled hypertension and at least one other cardiovascular disease risk factor treated at 30 primary care practices in Maryland and Pennsylvania. Fifteen practices randomized to the SCP arm receive standardized BP measurement training; race/ethnicity-specific audit and feedback of BP control rates; and quarterly webinars in management practices, quality improvement and disparities reduction. Fifteen practices in the CC/SC arm receive the SCP interventions plus implementation of the collaborative care model with stepped-care components (community health worker referrals and virtual specialist-panel consults). The primary clinical outcome is BP control (<140/90 mm Hg) at 12 months. The primary patient-reported outcome is change from baseline in self-reported patient activation at 12 months. DISCUSSION: This study will provide knowledge about the feasibility of leveraging existing resources in routine primary care and potential benefits of adding supportive community-facing roles to improve hypertension care and reduce disparities. TRIAL REGISTRATION: Clinicaltrials.govNCT02674464.

The Effects of Model Misspecification in Unanchored Matching-Adjusted Indirect Comparison: Results of a Simulation Study.

OBJECTIVES: To assess the performance of unanchored matching-adjusted indirect comparison (MAIC) by matching on first moments or higher moments in a cross-study comparisons under a variety of conditions. A secondary objective was to gauge the performance of the method relative to propensity score weighting (PSW). METHODS: A simulation study was designed based on an oncology example, where MAIC was used to account for differences between a contemporary trial in which patients had more favorable characteristics and a historical control. A variety of scenarios were then tested varying the setup of the simulation study, including violating the implicit or explicit assumptions of MAIC. RESULTS: Under ideal conditions and under a variety of scenarios, MAIC performed well (shown by a low mean absolute error [MAE]) and was unbiased (shown by a mean error [ME] of about zero). The performance of the method deteriorated where the matched characteristics had low explanatory power or there was poor overlap between studies. Only when important characteristics are not included in the matching did the method become biased (nonzero ME). Where the method showed poor performance, this was exaggerated if matching was also performed on the variance (ie, higher moments). Relative to PSW, MAIC provided similar results in most circumstances, although it exhibited slightly higher MAE and a higher chance of exaggerating bias. CONCLUSIONS: MAIC appears well suited to adjust for cross-trial comparisons provided the assumptions underpinning the model are met, with relatively little efficiency loss compared with PSW.