The "pragmatic trial": An essentially contested concept?

For over 50 years, clinical research methodology has wrestled with the problem of the lack of correspondence between tests of treatments and applications of treatments. The former comprise of trials featuring scrupulous control of patient eligibility, treatment compliance, clinician expertise, follow-up intensity, and so on. In applying a validated treatment, the practitioner has to confront considerable real-world variation in potential patients and in implementation regimes. The remedy, going by the name of "pragmatic trials," is to conduct clinical trials in conditions corresponding more closely to everyday practice. This solution has proved easier to utter than to execute, and the paper reviews the extensive literature on pragmatic trials, seeking to assess whether it has terminated in clarity or contestation.

Evaluation of bottom-up interventions targeting community-dwelling frail older people in Belgium: methodological challenges and lessons for future comparative effectiveness studies.

BACKGROUND: Optimizing the organization of care for community-dwelling frail older people is an important issue in many Western countries. In Belgium, a series of complex, innovative, bottom-up interventions was recently designed and implemented to help frail older people live at home longer. As the effectiveness of these interventions may vary between different population groups according to their long-term care needs, they must be evaluated by comparison with a control group that has similar needs. METHODS: The goal was to identify target groups for these interventions and to establish control groups with similar needs and to explore, per group, the extent to which the utilization of long-term care is matched to needs. We merged two databases: a clinical prospective database and the routine administrative database for healthcare reimbursements. Through Principal Component Analysis followed by Clustering, the intervention group was first stratified into disability profiles. Per profile, comparable control groups for clinical variables were established, based on propensity scores. Using chi-squared tests and logistic regression analysis, long-term care utilization at baseline was then compared per profile and group studied. RESULTS: Stratification highlighted five disability profiles: people with low-level limitations; people with limitations in instrumental activities of daily life and low-level of cognitive impairment; people with functional limitations; people with functional and cognitive impairments; and people with functional, cognitive, and behavioral problems. These profiles made it possible to identify long-term care needs. For instance, at baseline, those who needed more assistance with hygiene tasks also received more personal nursing care (P < 0.05). However, there were some important discrepancies between the need for long-term care and its utilization: while 21% of patients who were totally dependent for hygiene tasks received no personal nursing care, personal nursing care was received by 33% of patients who could perform hygiene tasks. CONCLUSIONS: The disability profiles provide information on long-term care needs but not on the extent to which those needs are met. To assess the effectiveness of interventions, controls at baseline should have similar disability profiles and comparable long-term care utilization. To allow for large comparative effectiveness studies, these dimensions should ideally be available in routine databases.

Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research.

BACKGROUND: The Core Outcome Measures in Effectiveness Trials (COMET) database is a publically available, searchable repository of published and ongoing core outcome set (COS) studies. An annual systematic review update is carried out to maintain the currency of database content. METHODS: The methods used in the fourth update of the systematic review followed the same approach used in the original review and previous updates. Studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Searches were carried out in March 2018 to identify studies that had been published or indexed between January 2017 and the end of December 2017. RESULTS: Forty-eight new studies, describing the development of 56 COS, were included. There has been an increase in the number of studies clearly specifying the scope of the COS in terms of the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics. Public participation has continued to rise with over half (n = 27, 56%) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased, in particular participation from non-clinical research experts has risen from 32% (mean average in previous reviews) to 62% (n = 29). Input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) have all increased since the previous reviews. CONCLUSION: This update included a pronounced increase in the number of new COS identified compared to the previous three updates. There was an improvement in the reporting of the scope, stakeholder participants and methods used. Furthermore, there has been an increase in participation from Australasia, Asia, South America and Africa. These advancements are reflective of the efforts made in recent years to raise awareness about the need for COS development and uptake, as well as developments in COS methodology.

Choosing important health outcomes for comparative effectiveness research: An updated systematic review and involvement of low and middle income countries.

BACKGROUND: Core outcome sets (COS) comprise a minimum set of outcomes that should be measured and reported in all trials for a specific health condition. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative maintains an up to date, publicly accessible online database of published and ongoing COS. An annual systematic review update is an important part of this process. METHODS: This review employed the same, multifaceted approach that was used in the original review and the previous two updates. This approach has identified studies that sought to determine which outcomes/domains to measure in clinical trials of a specific condition. This update includes an analysis of the inclusion of participants from low and middle income countries (LMICs) as identified by the OECD, in these COS. RESULTS: Eighteen publications, relating to 15 new studies describing the development of 15 COS, were eligible for inclusion in the review. Results show an increase in the use of mixed methods, including Delphi surveys. Clinical experts remain the most common stakeholder group involved. Overall, only 16% of the 259 COS studies published up to the end of 2016 have included participants from LMICs. CONCLUSION: This review highlights opportunities for greater public participation in COS development and the involvement of stakeholders from a wider range of geographical settings, in particular LMICs.

Nine years of comparative effectiveness research education and training: initiative supported by the PhRMA Foundation.

The term comparative effectiveness research (CER) took center stage with passage of the American Recovery and Reinvestment Act (2009). The companion US$1.1 billion in funding prompted the launch of initiatives to train the scientific workforce capable of conducting and using CER. Passage of the Patient Protection and Affordable Care Act (2010) focused these initiatives on patients, coining the term 'patient-centered outcomes research' (PCOR). Educational and training initiatives were soon launched. This report describes the initiative of the Pharmaceutical Research and Manufacturers Association of America (PhRMA) Foundation. Through provision of grant funding to six academic Centers of Excellence, to spearheading and sponsoring three national conferences, the PhRMA Foundation has made significant contributions to creation of the scientific workforce that conducts and uses CER/PCOR.

Registry-based randomized controlled trials- what are the advantages, challenges, and areas for future research?

Registry-based randomized controlled trials are defined as pragmatic trials that use registries as a platform for case records, data collection, randomization, and follow-up. Recently, the application of registry-based randomized controlled trials has attracted increasing attention in health research to address comparative effectiveness research questions in real-world settings, mainly due to their low cost, enhanced generalizability of findings, rapid consecutive enrollment, and the potential completeness of follow-up for the reference population, when compared with conventional randomized effectiveness trials. However several challenges of registry-based randomized controlled trials have to be taken into consideration, including registry data quality, ethical issues, and methodological challenges. In this article, we summarize the advantages, challenges, and areas for future research related to registry-based randomized controlled trials.

Ability and Use of Comparative Effectiveness Research by P&T Committee Members and Support Staff: A 1-Year Follow-up.

BACKGROUND: In recent years, comparative effectiveness tools and methods have evolved to assist health care decision makers in identifying optimal therapies. In-person training programs on comparative effectiveness research may be helpful in understanding and applying this information. OBJECTIVE: To provide a follow-up assessment of the use of comparative effectiveness research (CER) in the pharmacy and therapeutics (P&T) committee decision-making process, using information collected from participants 1 year after attending a live continuing education program, in which participants were taught about CER designs and how to access available CER resources through the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care (EHC) Program. METHODS: A retrospective, cross-sectional questionnaire was developed and mailed to 2 groups of individuals: CER workshop attendees and interested nonattendees (expressing an interest in attending a workshop but did not attend for unknown reasons). The questionnaire asked respondents to indicate personal and organizational use of CER in the decision-making process. Participants were asked to indicate whether their knowledge, ability, and use of CER studies increased since participating in the program. Data were analyzed using nonparametric tests to compare the responses of attendees and nonattendees, as well as overall reliability of the instrument. RESULTS: A total of 164 respondents completed the questionnaire (63 attendees and 101 nonattendees; overall response rate = 44%). The majority of respondents were pharmacists (n = 157, 95.7%) and were affiliated with a hospital (n = 106, 64.6%). Proportions of attendees and nonattendees differed significantly in the use of EHC research reviews/reports (45% and 28%, respectively; P = 0.02) and EHC executive summaries of research reviews/reports (48% and 29%, respectively; P = 0.01). At 1-year follow-up, the majority of attendees reported an increase ("somewhat" or "very much") in knowledge of CER (91.5%), ability to use CER (83.0%), and use of CER studies (58.7%). CONCLUSIONS: Health professionals attending a continuing education CER program reported higher use of EHC CER materials compared with nonattendees. Additionally, attendees reported increased use of CER in clinical decision making. A continuing education program such as this may provide an effective avenue for introducing CER methods and resources to the P&T committee and clinical decision-making processes. DISCLOSURES: This project was supported by grant number R18HS019220 from the Agency of Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency of Healthcare Research and Quality. Augustine has stock in Pfizer. The authors declared no other potential conflicts of interest associated with this study. Study concept and design were primarily contributed by Malone, along with Warholak, Hines, Brown, Hurwitz, and Taylor. Warholak, Hines, Brown, Hurwitz, and Taylor collected the data, assisted by Malone, Brixner, Cobaugh, and Schlaifer. Data interpretation was performed by Malone and Augustine, with assistance from the other authors. The manuscript was written primarily by Augustine, with assistance from Malone, Sun, Warholak, Hines, Brown, Hurwitz, and Taylor. Malone revised the manuscript, assisted by Warholak, Hines, Brown, Hurwitz, Taylor, Brixner, Cobaugh, and Schlaifer.

Got CER? Educating Pharmacists for Practice in the Future: New Tools for New Challenges.

BACKGROUND: Understanding how treatments work in the real world and in real patients is an important and complex task. In recent years, comparative effectiveness research (CER) studies have become more available for health care providers to inform evidence-based decision making. There is variability in the strengths and limitations of this new evidence, and researchers and decision makers are faced with challenges when assessing the quality of these new methods and CER studies. OBJECTIVES: To (a) describe an online tool developed by the CER Collaborative, composed of the Academy of Managed Care Pharmacy, the International Society for Pharmacoeconomics and Outcomes Research, and the National Pharmaceutical Council, and (b) provide an early evaluation of the training program impact on learners' self-reported abilities to evaluate and incorporate CER studies into their decision making. METHODS: To encourage greater transparency, consistency, and uniformity in the development and assessment of CER studies, the CER Collaborative developed an online tool to assist researchers, new and experienced clinicians, and decision makers in producing and evaluating CER studies. A training program that supports the use of the online tool was developed to improve the ability and confidence of individuals to apply CER study findings in their daily work. Seventy-one health care professionals enrolled in 3 separate cohorts for the training program. Upon completion, learners assessed their abilities to interpret and apply findings from CER studies by completing on online evaluation questionnaire. RESULTS: The first 3 cohorts of learners to complete the training program consisted of 71 current and future health care practitioners and researchers. At completion, learners indicated high confidence in their CER evidence assessment abilities (mean = 4.2). Learners reported a 27.43%-59.86% improvement in capabilities to evaluate various CER studies and identify study design flaws (mean evaluation before CER Certificate Program [CCP] scores = 1.86-3.14 and post-CCP scores = 3.92-4.24). Additionally, 63% of learners indicated that they expected to increase their use of evidence from CER studies in at least 1-2 problem decisions per month. CONCLUSIONS: The CER Collaborative has responded to the need for increased practitioner training to improve understanding and application of new CER studies. The CER Collaborative tool and certificate training program are innovative solutions to help decision makers meet the challenges they face in honing their skills to best incorporate credible and relevant CER evidence into their decision making. DISCLOSURES: The CER Collaborative, the development of the questionnaires and web-based tool, and the development of the CER Certificate Program were supported by grants and in-kind contributions from the Academy of Managed Care Pharmacy (AMCP), the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), and the National Pharmaceutical Council (NPC). The University of Maryland School of Pharmacy conducted its work under a contract with the AMCP Foundation and grant funding from the NPC. Perfetto is employed by the University of Maryland and the National Health Council and serves as assistant editor for the Journal of Managed Care & Specialty Pharmacy, consults for Avelere, and serves as a member of advisory boards for the PQA and CMTP. Pickering received support from the NPC for activities related to this research. Eichelberger is employed by the Academy of Managed Care Pharmacy. Eichelberger and Graff are with the CER Collaborative. Graff is employed by the National Pharmaceutical Council. Study concept and design were primarily contributed by Perfetto, Graff, and Eichelberger, along with Anyanwu and assisted by Pickering and Ward Zaghab. Pickering and Ward Zaghab took the lead in data collection, with assistance from the other authors, and data interpretation was performed by Perfetto, Graff, Pickering, and Ward Zaghab, with assistance from the other authors. The manuscript was written by Perfetto and Anyanwu, with assistance from the other authors, and revised by Graff, Perfetto, Anyanwu, and Pickering, assisted by Eichelberger and Ward Zaghab.

IMPACCT Kids' Care: a real-world example of stakeholder involvement in comparative effectiveness research.

The Patient-Centered Outcomes Research Institute has accelerated conversations about the importance of actively engaging stakeholders in all aspects of comparative effectiveness research (CER). Other scientific disciplines have a history of stakeholder engagement, yet few empirical examples exist of how these stakeholders can inform and enrich CER. Here we present a case study which includes the methods used to engage stakeholders, what we learned from them, and how we incorporated their ideas in a CER project. We selected stakeholders from key groups, built relationships with them and collected their feedback through interviews, observation and ongoing meetings during the four research process phases: proposal development, adapting study methods, understanding the context and information technology tool design and refinement.

Traumatic Brain Injury Rehabilitation Comparative Effectiveness Research: Introduction to the Traumatic Brain Injury-Practice Based Evidence Archives Supplement.

This supplement of the Archives of Physical Medicine and Rehabilitation is devoted to the Traumatic Brain Injury-Practice Based Evidence study, the first practice-based evidence study, to our knowledge, of traumatic brain injury rehabilitation. The purpose of this preface is to place this study in the broader context of comparative effectiveness research and introduce the articles in the supplement.

Futurescapes: evidence expectations in the USA for comparative effectiveness research for drugs in 2020.

AIM: Explore key factors influencing future expectations for the production of evidence from comparative effectiveness research for drugs in the USA in 2020 and construct three plausible future scenarios. MATERIALS & METHODS: Semistructured key informant interviews and three rounds of modified Delphi with systematic scenario-building methods. RESULTS & CONCLUSION: Most influential key factors were: health delivery system integration; electronic health record development; exploitation of very large databases and mixed data sources; and proactive patient engagement in research. The scenario deemed most likely entailed uneven development of large integrated health systems with pockets of increased provider risk for patient care, enhanced data collection systems, changing incentives to do comparative effectiveness research and new opportunities for evidence generation partnerships.

Optimizing the leveraging of real-world data to improve the development and use of medicines.

Health research, including health outcomes and comparative effectiveness research, is on the cusp of a golden era of access to digitized real-world data, catalyzed by the adoption of electronic health records and the integration of clinical and biological information with other data. This era promises more robust insights into what works in health care. Several barriers, however, will need to be addressed if the full potential of these new data are fully realized; these will involve both policy solutions and stakeholder cooperation. Although a number of these issues have been widely discussed, we focus on the one we believe is the most important-the facilitation of greater openness among public and private stakeholders to collaboration, connecting information and data sharing, with the goal of making robust and complete data accessible to all researchers. In this way, we can better understand the consequences of health care delivery, improve the effectiveness and efficiency of health care systems, and develop advancements in health technologies. Early real-world data initiatives illustrate both potential and the need for future progress, as well as the essential role of collaboration and data sharing. Health policies critical to progress will include those that promote open source data standards, expand access to the data, increase data capture and connectivity, and facilitate communication of findings.

SUPPORT and the Ethics of Study Implementation: Lessons for Comparative Effectiveness Research from the Trial of Oxygen Therapy for Premature Babies.

The design of SUPPORT has been widely misunderstood. This confusion has driven much of the debate about the trial - and threatens the whole enterpise of comparative effectiveness research.

Were There "Additional Foreseeable Risks" in the SUPPORT Study? Lessons Not Learned from the ARDSnet Clinical Trials.

Even though the interventions were adapted from standard clinical practice, the way they were provided meant that the care given infants in the study was distinctly different from standard care, with different risk profiles. Parents should have been informed about those differences.

Investigation of comparative effectiveness research in Asia, Europe, and North America.

Comparative effectiveness research (CER) is an important branch of pharmacoeconomics that systematically studies and evaluates the cost-effectiveness of medical interventions. CER plays instrumental roles in guiding government public health policy programs and insurance. Countries throughout the world use different methods of CER to help make medical decisions based on providing optimal therapy at a reduced cost. Expenses to the healthcare system continue to rise, and CER is one-way in which expenses could be curbed in the future by applying cost-effectiveness evidence to clinical decisions. China, India, South Korea, and the United Kingdom are of essential focus because these country's economies and health care expenses continue to expand. The structures and use of CER are diverse throughout these countries, and each is of prime importance. By conducting this thorough comparison of CER in different nations, strategies and organizational setups from different countries can be applied to help guide public health and medical decision-making in order to continue to expand the establishment and role of CER programs. The patient-centered medical home has been created to help reduce costs in the primary care sector and to help improve the effectiveness of therapy. Barriers to CER are also important as many stakeholders need to be able to work together to provide the best CER evidence. The advancement of CER in multiple countries throughout the world provides a possible way of reducing costs to the healthcare system in an age of expanding expenses.

Building electronic data infrastructure for comparative effectiveness research: accomplishments, lessons learned and future steps.

There are large gaps in our knowledge on the potential impact of diagnostics and therapeutics on outcomes of patients treated in the real world. Comparative effectiveness research aims to fill these gaps to maximize effectiveness of these interventions. Health information technology has the potential to dramatically improve the practice of medicine and of research. This is an overview of about US$100 million of American Recovery and Reinvestment Act investment in 12 projects managed by the Agency for Healthcare Research and Quality to build an electronic clinical data infrastructure that connects research with healthcare delivery. The achievements and lessons learned from these projects provided a foundation for the National Patient-Centered Clinical Research Network (PCORnet)and will help to guide future infrastructure development needed to build an efficient, scalable and sustainable learning health system.

Lessons from comparative effectiveness research methods development projects funded under the Recovery Act.

BACKGROUND: The American Recovery and Reinvestment Act of 2009 (ARRA) directed nearly US$29.2 million to comparative effectiveness research (CER) methods development. AIM: To help inform future CER methods investments, we describe the ARRA CER methods projects, identify barriers to this research and discuss the alignment of topics with published methods development priorities. METHODS: We used several existing resources and held discussions with ARRA CER methods investigators. RESULTS & CONCLUSION: Although funded projects explored many identified priority topics, investigators noted that much work remains. For example, given the considerable investments in CER data infrastructure, the methods development field can benefit from additional efforts to educate researchers about the availability of new data sources and about how best to apply methods to match their research questions and data.