Cost-sharing and adherence, clinical outcomes, health care utilization, and costs: A systematic literature review.

BACKGROUND: US health plans are adopting benefit designs that shift greater financial burden to patients through higher deductibles, additional copay tiers, and coinsurance. Prior systematic reviews found that higher cost was associated with reductions in both appropriate and inappropriate medications. However, these reviews were conducted prior to contemporary benefit design and medication utilization. OBJECTIVE: To assess the relationship and factors associated with cost-sharing and (1) medication adherence, (2) clinical outcomes, (3) health care resource utilization (HRU), and (4) costs. METHODS: A systematic review of literature published between January 2010 and August 2020 was conducted to identify the relationship between cost-sharing and medication adherence, clinical outcomes, HRU, and health care costs. Data were extracted using a standardized template and were synthesized by key questions of interest. RESULTS: From 1,995 records screened, 79 articles were included. Most studies, 71 of 79 (90%), reported the relationship between cost-sharing and treatment adherence, persistence and/or discontinuation; 16 (20%) reported data on cost-sharing and HRU or medication initiation, 11 (14%) on costsharing and health care costs, and 6 (8%) on cost-sharing and clinical outcomes. The majority of publications found that, regardless of disease area, increased cost-sharing was associated with worse adherence, persistence, or discontinuation. The aggregate data suggested the greater the magnitude of cost-sharing, the worse the adherence. Among studies examining clinical outcomes, cost-sharing was associated with worse outcomes in 1 study and the remaining 3 found no significant differences. Regarding HRU, higher-cost-sharing trended toward decreased outpatient and increased inpatient utilization. The available evidence suggested higher cost-sharing has an overall neutral to negative impact on total costs. Studies evaluating elimination of copays found either decreased or no impact in total costs. CONCLUSIONS: The published literature shows consistent impacts of higher cost sharing on initiation and continuation of medications, and the greater the cost-sharing, the worse the medication adherence. The evidence is limited regarding the impact of cost-sharing on clinical outcomes, HRU, and costs. Limited evidence suggests increased cost-sharing is associated with more inpatient care and less outpatient care; however, a neutral to no difference was suggested for other outcomes. Although increased costsharing is intended to decrease total costs, studies evaluating reducing or eliminating cost-sharing found that total costs did not rise. Today's growing cost-containment environment should carefully consider the broader impact cost-sharing has on treatment adherence, clinical outcomes, resource use, and total costs. It may be that cost-sharing is a blunt, rather than precise, tool to curb health care costs, affecting both necessary and unnecessary health care use. DISCLOSURES: This study and the development of this article were funded by the National Pharmaceutical Council. Mr Sils is an employee of the National Pharmaceutical Council. Dr Graff is a former employee of the National Pharmaceutical Council. Drs Fusco and Kistler and Ms Ruiz are employees of Xcenda. Xcenda received funding to conduct the literature review.

Prevalence of Avoidable and Bias-Inflicting Methodological Pitfalls in Real-World Studies of Medication Safety and Effectiveness.

Many real-word evidence (RWE) studies that utilize existing healthcare data to evaluate treatment effects incur substantial but avoidable bias from methodologically flawed study design; however, the extent of preventable methodological pitfalls in current RWE is unknown. To characterize the prevalence of avoidable methodological pitfalls with potential for bias in published claims-based studies of medication safety or effectiveness, we conducted an English-language search of PubMed for articles published from January 1, 2010 to May 20, 2019 and randomly selected 75 studies (10 case-control and 65 cohort studies) that evaluated safety or effectiveness of cardiovascular, diabetes, or osteoporosis medications using US health insurance claims. General and methodological study characteristics were extracted independently by two reviewers, and potential for bias was assessed across nine bias domains. Nearly all studies (95%) had at least one avoidable methodological issue known to incur bias, and 81% had potentially at least one of the four issues considered major due to their potential to undermine study validity: time-related bias (57%), potential for depletion of outcome-susceptible individuals (44%), inappropriate adjustment for postbaseline variables (41%), or potential for reverse causation (39%). The median number of major issues per study was 2 (interquartile range (IQR), 1-3) and was lower in cohort studies with a new-user, active-comparator design (median 1, IQR 0-1) than in cohort studies of prevalent users with a nonuser comparator (median 3, IQR 3-4). Recognizing and avoiding known methodological study design pitfalls could substantially improve the utility of RWE and confidence in its validity.

Limited role of patient input in specialty drug coverage policies.

BACKGROUND: Despite increased financial contributions towards care, consumers' role in shaping their insurance benefits is unclear. OBJECTIVE: To examine the role played by patient input when US commercial health plans formulate specialty drug coverage policies, along with the benefits and challenges of considering this input. METHODS: We employed a parallel, mixed-methods approach. First, we reviewed health plans' policy development processes as reported on their websites. Second, we reviewed a data set of private health plan coverage decisions for specialty drugs and examined whether the evidence cited in policies included patient-reported outcomes (eg, health-related quality of life endpoints) and patient-based methodological designs (eg, interviews or surveys of patients). Third, we performed a survey (N = 21 respondents) and interviews (N = 5 interviewees) with plan decision-makers to determine the current role of patient input in plan decision-making, and the benefits and challenges of incorporating this data when formulating specialty drug coverage policies. RESULTS: We found that plans do not commonly solicit patient input when developing coverage policies, with only two instances of limited interaction between plans and patients or members. 1,316 (9%) of the studies plans cited in their specialty drug coverage policies included at least one patient-reported endpoint, and 0.4% (N = 62) used a patient-based methodological design. Of studies with patient-based designs, 40 used interviews, 26 included surveys/questionnaires, and one concerned shared decision-making (design categories not mutually exclusive). Almost half of the survey respondents reported having never engaged with patients or members when developing coverage policies. Among respondents who had engaged with patients or members, most reported doing so only rarely. The survey and interviews highlighted various benefits of soliciting patient input, including the value of obtaining a humanistic perspective, and several challenges, including resource requirements and the quality of obtained information. CONCLUSIONS: We found a notable lack of patient and member engagement by commercial health plans when formulating drug coverage policies. Survey respondents and interviewees identified benefits of accounting for patients' and plan members' values and preferences in specialty drug coverage policies, but also reported a number of important challenges to doing so. DISCLOSURES: National Pharmaceutical Council provided funding for this research.

Stakeholders find that step therapy should be evidence-based, flexible, and transparent: assessing appropriateness using a consensus approach.

BACKGROUND: Step therapy, one approach to utilization management, is used by health plans to ensure safe and clinically appropriate care while managing cost. Several patient and provider groups have each developed principles to guide the appropriate use of step therapy; however, no comprehensive multistakeholder informed set of criteria exist. OBJECTIVE: To assess multistakeholder consensus on criteria for the development and implementation of step therapy for pharmaceutical therapies. Stakeholders were asked to (a) assess the appropriateness of step therapy as a utilization management tool; (b) rate specific criteria across 5 domains (development, implementation, communication, appeals, and evaluation) of step therapy; and (c) categorize these criteria as standards or best practices. METHODS: We conducted a multiphase project culminating in a roundtable of experts representing patient, provider, plan, pharmacy, policy, and ethical perspectives. We first reviewed guiding principles, position statements, and legislative activity to draft criteria regarding step therapy protocol development, implementation, communication, and evaluation. To assess consensus across a convenience sample of experts, we employed an iterative 4-step modified Delphi method. Panelists were asked to (a) rate the overall appropriateness of step therapy, (b) rate the appropriateness of specific criteria, and (c) identify each as a standard or best practice. Appropriateness was rated from 1-9 and categorized in terciles (1-3: not appropriate, 4-6: neither, 7-9: appropriate) to assess quantitative agreement, disagreement, and indeterminate agreement. RESULTS: After the second round of voting, roundtable panelists (n = 16) disagreed on the appropriateness of step therapy for utilization management (50% appropriate, 31.25% neither, and 18.75% inappropriate). Agreement was achieved on 21 criteria across 5 themes (clinical criteria as the foundation for protocol development, implementation of protocols, transparency and communication of processes, navigation of the appeals process, and evaluation of health and administrative impact). Fourteen and seven criteria were categorized as standards and best practices, respectively. CONCLUSIONS: The stakeholders in this panel differed in their assessments of the appropriateness of step therapy but agreed regarding how these protocols should be developed, implemented, communicated, and evaluated. Most criteria were rated as standards that can be used by stakeholders when developing, implementing, and assessing step therapy processes today. DISCLOSURES: This study was funded by the National Pharmaceutical Council. Karmarkar was a fellow at the National Pharmaceutical Council and Duke-Margolis Center for Health Policy at the time this study was conducted. Dubois and Graff are employees of the National Pharmaceutical Council. This work was previously presented as a virtual poster during the AMCP 2020 eLearning Days, April 21-24, 2020.

Is Real-World Evidence Used in P&T Monographs and Therapeutic Class Reviews?

BACKGROUND: Payers are faced with making coverage and reimbursement decisions based on the best available evidence. Often these decisions apply to patient populations, provider networks, and care settings not typically studied in clinical trials. Treatment effectiveness evidence is increasingly available from electronic health records, registries, and administrative claims. However, little is known about when and what types of real-world evidence (RWE) studies inform pharmacy and therapeutic (P&T) committee decisions. OBJECTIVE: To evaluate evidence sources cited in P&T committee monographs and therapeutic class reviews and assess the design features and quality of cited RWE studies. METHODS: A convenience sample of representatives from pharmacy benefit management, health system, and health plan organizations provided recent P&T monographs and therapeutic class reviews (or references from such documents). Two investigators examined and grouped references into major categories (published studies, unpublished studies, and other/unknown) and multiple subcategories (e.g., product label, clinical trials, RWE, systematic reviews). Cited comparative RWE was reviewed to assess design features (e.g., population, data source, comparators) and quality using the Good ReseArch for Comparative Effectiveness (GRACE) Checklist. RESULTS: Investigators evaluated 565 references cited in 27 monographs/therapeutic class reviews from 6 managed care organizations. Therapeutic class reviews mostly cited published clinical trials (35.3%, 155/439), while single-product monographs relied most on manufacturer-supplied information (42.1%, 53/126). Published RWE comprised 4.8% (21/439) of therapeutic class review references, and none (0/126) of the monograph references. Of the 21 RWE studies, 12 were comparative and assessed patient care settings and outcomes typically not included in clinical trials (community ambulatory settings [10], long-term safety [8]). RWE studies most frequently were based on registry data (6), conducted in the United States (6), and funded by the pharmaceutical industry (5). GRACE Checklist ratings suggested the data and methods of these comparative RWE studies were of high quality. CONCLUSIONS: RWE was infrequently cited in P&T materials, even among therapeutic class reviews where RWE is more readily available. Although few P&T materials cited RWE, the comparative RWE studies were generally high quality. More research is needed to understand when and what types of real-world studies can more routinely inform coverage and reimbursement decisions. DISCLOSURES: This project was funded by the National Pharmaceutical Council. Hurwitz, Brown, Peters, and Malone have nothing to disclose. Graff is employed by the National Pharmaceutical Council Part of this study was presented as a poster presentation at the AMCP Managed Care & Specialty Pharmacy 2016 Annual Meeting; April 19-22, 2016; San Francisco, CA. Study concept and design were primarily contributed by Malone and Graff, along with Hurwitz and Brown. All authors participated in data collection, and data interpretation was performed by Malone, Hurwitz, and Graff, with assistance from Brown and Peters. The manuscript was written primarily by Hurwitz and Malone, along with Graff, Brown, and Peters, and revised by Malone, Brown, Peters, Hurwitz, and Graff.

Evolution of the AMCP Format for Formulary Submissions.

DISCLOSURES: No funding was required for this project. The authors are or have been members of the Format Executive Committee.

Underestimating the Value of an Intervention: The Case for Including Productivity in Value Assessments and Formulary Design.

BACKGROUND: Value assessment reports are increasingly being considered in health care coverage decisions. The inputs included and analytic methodologies underlying these reports should include all components of value. OBJECTIVE: To determine whether and how productivity was included in a value assessment, compare the incremental cost per quality-adjusted life-year (cost/QALY) estimates with and without productivity, assess if inclusion of productivity changed the value category and estimate the direction and magnitude of change. METHODS: We reviewed pharmaceutical value assessment reports published between March 2017 and July 2019 by the Institute for Clinical and Economic Review (ICER) to determine whether productivity was included and how it was reported (i.e., co-base case or scenario analysis). Within each report, we identified unique treatment comparisons for which modelers estimated an incremental cost/QALY. We categorized the incremental cost/QALY estimates using ICER's willingness-to-pay (WTP) categories and assessed if inclusion of productivity changed the value category (i.e., < $50,000/QALY). For reports that included 2 numerical estimates, we assessed the direction and magnitude of change when productivity was included. RESULTS: Of the 19 reports that evaluated pharmaceutical treatments, 18 (94.7%) included productivity. Two reports (11.1%) incorporated productivity in a co-base case analysis, and 16 included productivity in a scenario analysis. Across these 18 reports, there were 75 unique comparisons of pharmaceutical interventions. Across the 75 comparisons, 4 (5.3%), 3 (4.0%), 8 (10.6%), and 1 (1.3%) of the coverage decisions would change at the $50,000/QALY, $100,000/QALY, $150,000/QALY, and $500,000/QALY threshold, respectively. Sixty comparisons included 2 numerical cost/QALY estimates. The magnitude of change in the cost/QALY, after including productivity, ranged from -80.1% to 6.8%. The estimated value increased for 54 (72%), decreased for 5 (6.6%), and did not change for 1 (1.7%) of the comparisons. CONCLUSIONS: Value assessment should capture the range of costs and benefits of an intervention. The exclusion of productivity costs can alter, often underestimating, the assessment of value. This may affect coverage decisions-inclusion or exclusion from the insurance benefit-based on these assessments. Value assessment reports intended to be used for health care decision making should include productivity and elevate its visibility by using base-case analyses rather than scenario analyses. DISCLOSURES: This study was funded by the National Pharmaceutical Council. Karmarkar is currently a postdoctoral fellow at the National Pharmaceutical Council. Graff and Westrich are employees of the National Pharmaceutical Council, which provides unrestricted research grants to value assessment bodies including ICER and IVI.

Little Consistency In Evidence Cited By Commercial Plans For Specialty Drug Coverage.

We found wide variation in the evidence that US commercial health plans reported reviewing in their specialty drug coverage policies. There was little consistency in the numbers or types of studies cited by health plans. On average, only 15 percent of health plans' coverage policies cited the same study evaluating a specific drug for a specific indication.

Patient-Community Perspectives on Real-World Evidence: Enhancing Engagement, Understanding, and Trust.

BACKGROUND: Healthcare stakeholders have pronounced both enthusiasm and apprehension over the expanding use of real-world evidence (RWE). The patient community-those who benefit from new treatments but are vulnerable to potential safety risks and whose routine medical encounters are used to generate RWE-has been less vocal. Understanding patient perspectives on the use of RWE to guide clinical decision making and inform regulatory decisions and value assessments is imperative. METHODS: We convened a day-long, multi-stakeholder roundtable in Washington D.C., USA, on 31 July 2017 to gather patient-community views on RWE and related concerns and the communications, information and tools needed by patients to understand, trust, and use RWE. Participants included a convenience sample of National Health Council (NHC) members primarily representing patient groups as well as non-patient members with an interest in RWE. Participants were organized into small, pre-assigned groups, ensuring representativeness across stakeholders and patient leadership. Discussions, including storyboards, notes, and illustrative examples were captured and later analyzed thematically by NHC staff. RESULTS: Ten RWE themes emerged: (1) most patients were unaware of RWE and its actual or potential uses, (2) common definitions for real-world data and RWE are needed, (3) patient organizations need RWE skills and tools, (4) patient-scientist partnerships can help differentiate high-quality RWE, (5) RWE should inform decision making, (6) clinician support is needed for RWE uptake in patient decision making, (7) communications to patients should be balanced and empowering, (8) context of use impacts RWE acceptability/trust, (9) privacy/data ownership require clarity, and (10) patient-generated data are also real-world data (RWD). CONCLUSION: Patients see great possibility in using RWE to understand how a treatment works-to find someone that "looks like me" as assurance of how a treatment might benefit them personally. Patient groups will play a critical role in helping to educate constituents on understanding, contributing to, and using RWE. To maximize patient uptake and the co-development and application of RWE, patient groups require education and tools.

Specialty Drug Coverage Varies Across Commercial Health Plans In The US.

We analyzed specialty drug coverage decisions issued by the largest US commercial health plans to examine variation in coverage and the consistency of those decisions with indications approved by the Food and Drug Administration (FDA). Across 3,417 decisions, 16 percent of the 302 drug-indication pairs were covered the same way by all of the health plans, and 48 percent were covered the same way by 75 percent of the plans. Specifically, 52 percent of the decisions were consistent with the FDA label, 9 percent less restrictive, 2 percent mixed (less restrictive in some ways but more restrictive in others), and 33 percent more restrictive, while 5 percent of the pairs were not covered. Health plans restricted coverage of drugs indicated for cancer less often than they did coverage of drugs indicated for other diseases. Using multivariate regression, we found that several drug-related factors were associated with less restrictive coverage, including indications for orphan diseases or pediatric populations, absence of safety warnings, time on the market, lack of alternatives, and expedited FDA review. Variations in coverage have implications for patients' access to treatment and health system costs.

Real-World Evidence: Useful in the Real World of US Payer Decision Making? How? When? And What Studies?

OBJECTIVES: To examine how real-world evidence (RWE) is currently perceived and used in managed care environments, especially to inform pharmacy and therapeutic (P&T) committee decisions, to assess which study factors (e.g., data, design, and funding source) contribute to RWE utility in decisions, and to identify barriers to consideration of RWE studies in P&T decision making. METHODS: We conducted focus groups/telephone-based interviews and surveys to understand perceptions of RWE and assess awareness, quality, and relevance of two high-profile examples of published RWE studies. A purposive sample comprised 4 physicians, 15 pharmacists, and 1 researcher representing 18 US health plans and health system organizations. RESULTS: Participants reported that RWE was generally used, or useful, to inform safety monitoring, utilization management, and cost analysis, but less so to guide P&T decisions. Participants were not aware of the two sample RWE studies but considered both studies to be valuable. Relevant research questions and outcomes, transparent methods, study quality, and timely results contribute to the utility of published RWE. Perceived organizational barriers to the use of published RWE included lack of skill, training, and timely study results. CONCLUSIONS: Payers recognize the value of RWE, but use of such studies to inform P&T decisions varies from organization to organization and is limited. Relevance to payers, timeliness, and transparent methods were key concerns with RWE. Participants recognized the need for continuing education on evaluating and using RWE to better understand the study methods, findings, and applicability to their organizations.

PEER-REVIEWED JOURNAL EDITORS' VIEWS ON REAL-WORLD EVIDENCE.

OBJECTIVES: Peer-review publication is a critical step to the translation and dissemination of research results into clinical practice guidelines, health technology assessment (HTA) and payment policies, and clinical care. The objective of this study was to examine current views of journal editors regarding: (i) The value of real-world evidence (RWE) and how it compares with other types of studies; (ii) Education and/or resources journal editors provide to their peer reviewers or perceive as needed for authors, reviewers, and editors related to RWE. METHODS: Journal editors' views on the value of RWE and editorial procedures for RWE manuscripts were obtained through telephone interviews, a survey, and in-person, roundtable discussion. RESULTS: In total, seventy-nine journals were approached, resulting in fifteen telephone interviews, seventeen survey responses and eight roundtable participants. RWE was considered valuable by all interviewed editors (n = 15). Characteristics of high-quality RWE manuscripts included: novelty/relevance, rigorous methodology, and alignment of data to research question. Editors experience challenges finding peer reviewers; however, these challenges persist across all study designs. Journals generally do not provide guidance, assistance, or training for reviewers, including for RWE studies. Health policy/health services research (HSR) editors were more likely than specialty or general medicine editors to participate in this study, potentially indicating that HSR researchers are more comfortable/interested in RWE. CONCLUSIONS: Editors report favorable views of RWE studies provided studies examine important questions and are methodologically rigorous. Improving peer-review processes across all study designs, has the potential to improve the evidence base for decision making, including HTA.

Does a One-Size-Fits-All Cost-Sharing Approach Incentivize Appropriate Medication Use? A Roundtable on the Fairness and Ethics Associated with Variable Cost Sharing.

BACKGROUND: Tiered formularies, in which patients pay copays or coinsurance out-of-pocket (OOP), are used to manage costs and encourage more efficient health care resource use. Formulary tiers are typically based on the cost of treatment rather than the medical appropriateness for the patient. Cost sharing may have unintended consequences on treatment adherence and health outcomes. Use of higher-cost, higher-tier medications can be due to a variety of factors, including unsuccessful treatment because of lack of efficacy or side effects, patient clinical or genetic characteristics, patient preferences to avoid potential side effects, or patient preferences based on the route of administration. For example, patients with rheumatoid arthritis may be required to fail low-cost generic treatments before obtaining coverage for a higher-tier tumor necrosis factor alpha inhibitor for which they would have a larger financial burden. Little is known about stakeholders' views on the acceptability of greater patient cost sharing if the individual patient characteristics lead to the higher-cost treatments. OBJECTIVE: To identify and discuss the trade-offs associated with variable cost sharing in pharmacy benefits. METHODS: To discuss the trade-offs associated with variable cost sharing in pharmacy benefits, we convened an expert roundtable of patient, payer, and employer representatives (panelists). Panelists reviewed background white papers, including an ethics framework; actuarial analysis; legal review; and stakeholder perspectives representing health plan, employer, and patient views. Using case studies, panelists were asked to consider (a) when it would be more (or less) acceptable to require higher cost sharing; (b) the optimal distribution of financial burdens across patients, all plan members, and employers; and (c) the existing barriers and potential solutions to align OOP costs with medically appropriate treatments. RESULTS: Panelists felt it was least acceptable for patients to have greater OOP costs if the use of the higher-cost treatment was due to biological reasons such as step therapy (6 = unacceptable, 9 = neutral, 2 = acceptable) or diagnostic results (5 = unacceptable, 10 = neutral, and 2 = acceptable). In contrast, panelists felt it was more acceptable for patients to pay greater OOP costs when treatment choice was based on preferences to avoid a side-effect risk (1 = unacceptable, 3 = neutral, and 13 = acceptable) or the route/frequency of administration (1 = unacceptable, 1 = neutral, and 15 = acceptable). Five guiding principles emerged from the discussion: When patients have tried lower-cost therapies unsuccessfully, the benefits of higher-cost treatments were certain and significant, the cost difference between treatments was aligned with improved benefits, and penalties due to bad luck were mitigated, then cost-sharing differences should be minimized but not eliminated. CONCLUSIONS: Patient OOP costs can affect the use of both inappropriate and appropriate medications. This study identified 5 guiding principles to determine when it was more (or less) acceptable for patients with the same or similar conditions to have different OOP costs. Barriers that hinder the alignment of care and patient cost sharing exist. Policies that facilitate the alignment of patient cost sharing with appropriate care are needed. DISCLOSURES: Funding for this roundtable was provided by the National Pharmaceutical Council (NPC). Graff and Dubois are employed by the NPC. Shih was employed by the NPC at the time of this study. Barker, Dieguez, Sherman, and Larson received consulting fees for participation in this study. Larson also reports receiving grants and other payment from multiple major pharmaceutical manufacturers outside of this study. The NPC employees developed the study design and chose the case studies in collaboration with the white paper authors. The roundtable was facilitated by Dubois, and the meeting summary and manuscript were written by Graff and Shih, with revisions by all roundtable participants. The abstract for this article was previously presented as a poster at the following meetings: Stakeholder perspectives on balancing patient-centeredness and drug costs in the design of pharmacy benefits. Presented at: Academy of Managed Care Pharmacy 27th Annual Meeting & Expo; San Diego, California; April 8, 2015. Considering efficiency and fairness in the design of prescription drug benefits: seeking a balanced approach to improve patient access to medically appropriate medication and manage drug costs. Presented at: AcademyHealth Annual Research Meeting; Minneapolis, Minnesota; June 15, 2015. Study concept and design were contributed by Shih, Dubois, and Graff, along with Barker and Dieguez. Barker and Dieguez took the lead in data collection, assisted by Graff, Shih, and Dubois. Data interpretation was performed by Shih, Larson, Sherman, and Graff, with assistance from Dubois. The manuscript was written and revised by Graff and Shih, with assistance from the other authors.

Is Real-World Evidence Used in P&T Monographs and Therapeutic Class Reviews?

BACKGROUND: Payers are faced with making coverage and reimbursement decisions based on the best available evidence. Often these decisions apply to patient populations, provider networks, and care settings not typically studied in clinical trials. Treatment effectiveness evidence is increasingly available from electronic health records, registries, and administrative claims. However, little is known about when and what types of real-world evidence (RWE) studies inform pharmacy and therapeutic (P&T) committee decisions. OBJECTIVE: To evaluate evidence sources cited in P&T committee monographs and therapeutic class reviews and assess the design features and quality of cited RWE studies. METHODS: A convenience sample of representatives from pharmacy benefit management, health system, and health plan organizations provided recent P&T monographs and therapeutic class reviews (or references from such documents). Two investigators examined and grouped references into major categories (published studies, unpublished studies, and other/unknown) and multiple subcategories (e.g., product label, clinical trials, RWE, systematic reviews). Cited comparative RWE was reviewed to assess design features (e.g., population, data source, comparators) and quality using the Good ReseArch for Comparative Effectiveness (GRACE) Checklist. RESULTS: Investigators evaluated 565 references cited in 27 monographs/therapeutic class reviews from 6 managed care organizations. Therapeutic class reviews mostly cited published clinical trials (35.3%, 155/439), while single-product monographs relied most on manufacturer-supplied information (42.1%, 53/126). Published RWE comprised 4.8% (21/439) of therapeutic class review references, and none (0/126) of the monograph references. Of the 21 RWE studies, 12 were comparative and assessed patient care settings and outcomes typically not included in clinical trials (community ambulatory settings [10], long-term safety [8]). RWE studies most frequently were based on registry data (6), conducted in the United States (6), and funded by the pharmaceutical industry (5). GRACE Checklist ratings suggested the data and methods of these comparative RWE studies were of high quality. CONCLUSIONS: RWE was infrequently cited in P&T materials, even among therapeutic class reviews where RWE is more readily available. Although few P&T materials cited RWE, the comparative RWE studies were generally high quality. More research is needed to understand when and what types of real-world studies can more routinely inform coverage and reimbursement decisions. DISCLOSURES: This project was funded by the National Pharmaceutical Council. Hurwitz, Brown, Peters, and Malone have nothing to disclose. Graff is employed by the National Pharmaceutical Council Part of this study was presented as a poster presentation at the AMCP Managed Care & Specialty Pharmacy 2016 Annual Meeting; April 19-22, 2016; San Francisco, CA. Study concept and design were primarily contributed by Malone and Graff, along with Hurwitz and Brown. All authors participated in data collection, and data interpretation was performed by Malone, Hurwitz, and Graff, with assistance from Brown and Peters. The manuscript was written primarily by Hurwitz and Malone, along with Graff, Brown, and Peters, and revised by Malone, Brown, Peters, Hurwitz, and Graff.

Got CER? Educating Pharmacists for Practice in the Future: New Tools for New Challenges.

BACKGROUND: Understanding how treatments work in the real world and in real patients is an important and complex task. In recent years, comparative effectiveness research (CER) studies have become more available for health care providers to inform evidence-based decision making. There is variability in the strengths and limitations of this new evidence, and researchers and decision makers are faced with challenges when assessing the quality of these new methods and CER studies. OBJECTIVES: To (a) describe an online tool developed by the CER Collaborative, composed of the Academy of Managed Care Pharmacy, the International Society for Pharmacoeconomics and Outcomes Research, and the National Pharmaceutical Council, and (b) provide an early evaluation of the training program impact on learners' self-reported abilities to evaluate and incorporate CER studies into their decision making. METHODS: To encourage greater transparency, consistency, and uniformity in the development and assessment of CER studies, the CER Collaborative developed an online tool to assist researchers, new and experienced clinicians, and decision makers in producing and evaluating CER studies. A training program that supports the use of the online tool was developed to improve the ability and confidence of individuals to apply CER study findings in their daily work. Seventy-one health care professionals enrolled in 3 separate cohorts for the training program. Upon completion, learners assessed their abilities to interpret and apply findings from CER studies by completing on online evaluation questionnaire. RESULTS: The first 3 cohorts of learners to complete the training program consisted of 71 current and future health care practitioners and researchers. At completion, learners indicated high confidence in their CER evidence assessment abilities (mean = 4.2). Learners reported a 27.43%-59.86% improvement in capabilities to evaluate various CER studies and identify study design flaws (mean evaluation before CER Certificate Program [CCP] scores = 1.86-3.14 and post-CCP scores = 3.92-4.24). Additionally, 63% of learners indicated that they expected to increase their use of evidence from CER studies in at least 1-2 problem decisions per month. CONCLUSIONS: The CER Collaborative has responded to the need for increased practitioner training to improve understanding and application of new CER studies. The CER Collaborative tool and certificate training program are innovative solutions to help decision makers meet the challenges they face in honing their skills to best incorporate credible and relevant CER evidence into their decision making. DISCLOSURES: The CER Collaborative, the development of the questionnaires and web-based tool, and the development of the CER Certificate Program were supported by grants and in-kind contributions from the Academy of Managed Care Pharmacy (AMCP), the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), and the National Pharmaceutical Council (NPC). The University of Maryland School of Pharmacy conducted its work under a contract with the AMCP Foundation and grant funding from the NPC. Perfetto is employed by the University of Maryland and the National Health Council and serves as assistant editor for the Journal of Managed Care & Specialty Pharmacy, consults for Avelere, and serves as a member of advisory boards for the PQA and CMTP. Pickering received support from the NPC for activities related to this research. Eichelberger is employed by the Academy of Managed Care Pharmacy. Eichelberger and Graff are with the CER Collaborative. Graff is employed by the National Pharmaceutical Council. Study concept and design were primarily contributed by Perfetto, Graff, and Eichelberger, along with Anyanwu and assisted by Pickering and Ward Zaghab. Pickering and Ward Zaghab took the lead in data collection, with assistance from the other authors, and data interpretation was performed by Perfetto, Graff, Pickering, and Ward Zaghab, with assistance from the other authors. The manuscript was written by Perfetto and Anyanwu, with assistance from the other authors, and revised by Graff, Perfetto, Anyanwu, and Pickering, assisted by Eichelberger and Ward Zaghab.

Data, Data Everywhere, but Access Remains a Big Issue for Researchers: A Review of Access Policies for Publicly-Funded Patient-Level Health Care Data in the United States.

INTRODUCTION: High quality research regarding treatment effectiveness, quality, and value is critical for improving the U.S. health care system. Recognition of this has led federal and state officials to better leverage existing data sources such as medical claims and survey data, but access must be balanced with privacy concerns. METHODS: We reviewed and catalogued data access policies for a selection of publicly-funded federal and state datasets to investigate how such policies may be promoting or limiting research activities. RESULTS: We found significant variation in data access policies across federal agencies and across state agencies, including variation for multiple datasets available from the same agency. We also observed numerous indirect hurdles to use of data, including complex data use application procedures, high user fees, and prolonged wait times for data delivery. CONCLUSIONS: Policy makers and data owners should consider making changes to data access policies to maximize the utility and availability of these valuable resources.

Standards and guidelines for observational studies: quality is in the eye of the beholder.

OBJECTIVES: Patient care decisions demand high-quality research. To assist those decisions, numerous observational studies are being performed. Are the standards and guidelines to assess observational studies consistent and actionable? What policy considerations should be considered to ensure decision makers can determine if an observational study is of high-quality and valid to inform treatment decisions? STUDY DESIGN AND SETTING: Based on a literature review and input from six experts, we compared and contrasted nine standards/guidelines using 23 methodological elements involved in observational studies (e.g., study protocol, data analysis, and so forth). RESULTS: Fourteen elements (61%) were addressed by at least seven standards/guidelines; 12 of these elements disagreed in the approach. Nine elements (39%) were addressed by six or fewer standards/guidelines. Ten elements (43%) were not actionable in at least one standard/guideline that addressed the element. CONCLUSION: The lack of observational study standard/guideline agreement may contribute to variation in study conduct; disparities in what is considered credible research; and ultimately, what evidence is adopted. A common set of agreed on standards/guidelines for conducting observational studies will benefit funders, researchers, journal editors, and decision makers.

Developing evidence that is fit for purpose: a framework for payer and research dialogue.

OBJECTIVES: Matching the supply and demand of evidence requires an understanding of when more evidence is needed, as well as the type of evidence that will meet this need. This article describes efforts to develop and refine a decision-making framework that considers payers' perspectives on the utility of evidence generated by different types of research methods, including real-world evidence. STUDY DESIGN: Conceptual framework development with subsequent testing during a roundtable dialogue. METHODS: The framework development process included a literature scan to identify existing frameworks and relevant articles on payer decision making. The framework was refined during a stand-alone roundtable in December 2013 hosted by the research team, which included representatives from public and private payers, pharmacy benefit management, the life sciences industry, and researchers. The roundtable discussion also included an application of the framework to 3 case studies. RESULTS: Application of the framework to the clinical scenarios and the resulting discussion provided key insights into when new evidence is needed to inform payer decision making and what questions should be addressed. Payers are not necessarily seeking more evidence about treatment efficacy; rather, they are seeking more evidence for relevant end points that illustrate the differences between treatment alternatives that can justify the resources required to change practice. In addition, payers are interested in obtaining new evidence that goes beyond efficacy, with an emphasis on effectiveness, longer-term safety, and delivery system impact. CONCLUSIONS: We believe that our decision-making framework is a useful tool to increase dialogue between evidence generators and payers, while also allowing for greater efficiency in the research process.

Individual treatment effects: implications for research, clinical practice, and policy.

Funding for comparative effectiveness research (CER) has focused attention on what treatments work best under what specific clinical circumstances, and for whom. Because not all patients respond in the same way, treatment decisions, clinical guidelines, and coverage policies applied in a "one-size-fits-all" fashion based upon the population "average" response may lead to suboptimal outcomes. Existing frameworks focus on why patients respond differently to treatments. We propose a framework that identifies when these differences are likely to be clinically important. Scenarios are presented in which it may be most critical for clinical decisions and policies to distinguish between the average and the individual patient so that treatment recommendations provide the greatest benefits for the largest number of patients. We provide recommendations for researchers to help identify issues to study, for providers to help assist them in recommending optimal treatment for individual patients, and for payers or public health bodies to help balance societal needs with those of the individual.

The good, the bad, and the different: a primer on aspects of heterogeneity of treatment effects.

The concept of heterogeneity is concerned with understanding differences within and across patients and studies. Heterogeneity of treatment effects is nonrandom variability in response to treatment and includes both benefits and harms. Because not all patients respond the same way, treatment decisions applied in a "one size fits all" fashion based on the average response observed in clinical trials may lead to suboptimal outcomes for some patients. Variation in outcomes among patients may be caused by observable and nonobservable factors. Changes in patients' health status over time can contribute to variability among patients. Assuming that the results from clinical trials are homogeneous across patients may fail to take into account clinically significant variability where some patients may receive benefit and others harm. Subgroup analyses and prediction models are 2 tools to explain variability observed within a study. Evidence synthesis with meta-analysis can provide useful information on the overall effectiveness and response among groups of patients undersampled in individual studies. Yet caution is warranted if the meta-analysis is missing studies or the individual studies comprising the meta-analysis are inherently different.For those making clinical, coverage, and reimbursement decisions at a population level, such as clinicians and pharmacy and therapeutics committee members, understanding the variation among patients, among subpopulations or populations of patients, among clinical studies, or within a meta-analysis is important to ensuring optimal patient outcomes. This article presents a variety of tools and resources to aid decision makers as they evaluate the literature to determine when clinically relevant differences exist.