Enabling Open Science for Health Research: Collaborative Informatics Environment for Learning on Health Outcomes (CIELO).

BACKGROUND: There is an emergent and intensive dialogue in the United States with regard to the accessibility, reproducibility, and rigor of health research. This discussion is also closely aligned with the need to identify sustainable ways to expand the national research enterprise and to generate actionable results that can be applied to improve the nation's health. The principles and practices of Open Science offer a promising path to address both goals by facilitating (1) increased transparency of data and methods, which promotes research reproducibility and rigor; and (2) cumulative efficiencies wherein research tools and the output of research are combined to accelerate the delivery of new knowledge in proximal domains, thereby resulting in greater productivity and a reduction in redundant research investments. OBJECTIVES: AcademyHealth's Electronic Data Methods (EDM) Forum implemented a proof-of-concept open science platform for health research called the Collaborative Informatics Environment for Learning on Health Outcomes (CIELO). METHODS: The EDM Forum conducted a user-centered design process to elucidate important and high-level requirements for creating and sustaining an open science paradigm. RESULTS: By implementing CIELO and engaging a variety of potential users in its public beta testing, the EDM Forum has been able to elucidate a broad range of stakeholder needs and requirements related to the use of an open science platform focused on health research in a variety of "real world" settings. CONCLUSIONS: Our initial design and development experience over the course of the CIELO project has provided the basis for a vigorous dialogue between stakeholder community members regarding the capabilities that will add the greatest value to an open science platform for the health research community. A number of important questions around user incentives, sustainability, and scalability will require further community dialogue and agreement.

AcademyHealth's Delivery System Science Fellowship: Training Embedded Researchers to Design, Implement, and Evaluate New Models of Care.

AcademyHealth's Delivery System Science Fellowship (DSSF) provides a paid postdoctoral pragmatic learning experience to build capacity within learning healthcare systems to conduct research in applied settings. The fellowship provides hands-on training and professional leadership opportunities for researchers. Since its inception in 2012, the program has grown rapidly, with 16 health systems participating in the DSSF to date. In addition to specific projects conducted within health systems (and numerous publications associated with those initiatives), the DSSF has made several broader contributions to the field, including defining delivery system science, identifying a set of training objectives for researchers working in delivery systems, and developing a national collaborative network of care delivery organizations, operational leaders, and trainees. The DSSF is one promising approach to support higher-value care by promoting continuous learning and improvement in health systems.

Distributed Data Networks That Support Public Health Information Needs.

INTRODUCTION: Data networks, consisting of pooled electronic health data assets from health care providers serving different patient populations, promote data sharing, population and disease monitoring, and methods to assess interventions. Better understanding of data networks, and their capacity to support public health objectives, will help foster partnerships, expand resources, and grow learning health systems. METHODS: We conducted semistructured interviews with 16 key informants across the United States, identified as network stakeholders based on their respective experience in advancing health information technology and network functionality. Key informants were asked about their experience with and infrastructure used to develop data networks, including each network's utility to identify and characterize populations, usage, and sustainability. RESULTS: Among 11 identified data networks representing hundreds of thousands of patients, key informants described aggregated health care clinical data contributing to population health measures. Key informant interview responses were thematically grouped to illustrate how networks support public health, including (1) infrastructure and information sharing; (2) population health measures; and (3) network sustainability. CONCLUSION: Collaboration between clinical data networks and public health entities presents an opportunity to leverage infrastructure investments to support public health. Data networks can provide resources to enhance population health information and infrastructure.

Framework To Guide The Collection And Use Of Patient-Reported Outcome Measures In The Learning Healthcare System.

BACKGROUND: Web-based collection of patient-reported outcome measures (PROMs) in clinical practice is expanding rapidly as electronic health records include web portals for patients to report standardized assessments of their symptoms. As the value of PROMs in patient care expands, a framework to guide the implementation planning, collection, and use of PROs to serve multiple goals and stakeholders is needed. METHODS: We identified diverse clinical, quality, and research settings where PROMs have been successfully integrated into care and routinely collected and analyzed drivers of successful implementation. Findings are based on key informant interviews with 46 individuals representing 38 organizations, of whom 40 participated in a webinars series, and 25 attended an in-person workshop designed to enable broad stakeholder input, review and refinement of the proposed PROMs implementation model. Stakeholders identified differing uses of PROMs to support: 1) individual patient care decisions, 2) quality improvement initiatives, 3) payer mandates, and 4) population health and research. RESULTS: The implementation framework and steps that are consistently identified by stakeholders as best practices to guide PROM capture and use are described. Of note, participants indicate that web-based informatics tools are necessary but not sufficient for PROM use, suggesting that successful PROM implementation requires integration into clinic operations and careful planning for user's analytic needs. Each of the four identified uses may require implementation modifications at each step to assure optimal use. CONCLUSIONS: The proposed framework will guide future PROM implementation efforts across learning health care systems to assure that complete PROMs are captured at the correct time, and with associated risk factors, to generate meaningful information to serve diverse stakeholders.

A Framework for Classification of Electronic Health Data Extraction-Transformation-Loading Challenges in Data Network Participation.

BACKGROUND: Contributing health data to national, regional, and local networks or registries requires data stored in local systems with local structures and codes to be extracted, transformed, and loaded into a standard format called a Common Data Model (CDM). These processes called Extract, Transform, Load (ETL) require data partners or contributors to invest in costly technical resources with specialized skills in data models, terminologies, and programming. Given the wide range of tasks, skills, and technologies required to transform data into a CDM, a classification of ETL challenges can help identify needed resources, which in turn may encourage data partners with less-technical capabilities to participate in data-sharing networks. METHODS: We conducted key-informant interviews with data partner representatives to survey the ETL challenges faced in clinical data research networks (CDRNs) and registries. A list of ETL challenges, organized into six themes was vetted during a one-day workshop with a wide range of network stakeholders including data partners, researchers, and policy experts. RESULTS: We identified 24 technical ETL challenges related to the data sharing process. All of these ETL challenges were rated as "important" or "very important" by workshop participants using a five point Likert scale. Based on these findings, a framework for categorizing ETL challenges according to ETL phases, themes, and levels of data network participation was developed. CONCLUSIONS: Overcoming ETL technical challenges require significant investments in a broad array of information technologies and human resources. Identifying these technical obstacles can inform optimal resource allocation to minimize the barriers and cost of entry for new data partners into extant networks, which in turn can expand data networks' inclusiveness and diversity. This paper offers pertinent information and guiding framework that are relevant for data partners in ascertaining challenges associated with contributing data in data networks.

A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data.

OBJECTIVE: Harmonized data quality (DQ) assessment terms, methods, and reporting practices can establish a common understanding of the strengths and limitations of electronic health record (EHR) data for operational analytics, quality improvement, and research. Existing published DQ terms were harmonized to a comprehensive unified terminology with definitions and examples and organized into a conceptual framework to support a common approach to defining whether EHR data is 'fit' for specific uses. MATERIALS AND METHODS: DQ publications, informatics and analytics experts, managers of established DQ programs, and operational manuals from several mature EHR-based research networks were reviewed to identify potential DQ terms and categories. Two face-to-face stakeholder meetings were used to vet an initial set of DQ terms and definitions that were grouped into an overall conceptual framework. Feedback received from data producers and users was used to construct a draft set of harmonized DQ terms and categories. Multiple rounds of iterative refinement resulted in a set of terms and organizing framework consisting of DQ categories, subcategories, terms, definitions, and examples. The harmonized terminology and logical framework's inclusiveness was evaluated against ten published DQ terminologies. RESULTS: Existing DQ terms were harmonized and organized into a framework by defining three DQ categories: (1) Conformance (2) Completeness and (3) Plausibility and two DQ assessment contexts: (1) Verification and (2) Validation. Conformance and Plausibility categories were further divided into subcategories. Each category and subcategory was defined with respect to whether the data may be verified with organizational data, or validated against an accepted gold standard, depending on proposed context and uses. The coverage of the harmonized DQ terminology was validated by successfully aligning to multiple published DQ terminologies. DISCUSSION: Existing DQ concepts, community input, and expert review informed the development of a distinct set of terms, organized into categories and subcategories. The resulting DQ terms successfully encompassed a wide range of disparate DQ terminologies. Operational definitions were developed to provide guidance for implementing DQ assessment procedures. The resulting structure is an inclusive DQ framework for standardizing DQ assessment and reporting. While our analysis focused on the DQ issues often found in EHR data, the new terminology may be applicable to a wide range of electronic health data such as administrative, research, and patient-reported data. CONCLUSION: A consistent, common DQ terminology, organized into a logical framework, is an initial step in enabling data owners and users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary. Future work will leverage the framework and terminology to develop reusable data quality assessment and reporting methods.

Concordium 2015: Strategic Uses of Evidence to Transform Delivery Systems.

In September 2015 the EDM Forum hosted AcademyHealth's newest national conference, Concordium. The 11 papers featured in the eGEMs "Concordium 2015" special issue successfully reflect the major themes and issues discussed at the meeting. Many of the papers address informatics or methodological approaches to natural language processing (NLP) or text analysis, which is indicative of the importance of analyzing text data to gain insights into care coordination and patient-centered outcomes. Perspectives on the tools and infrastructure requirements that are needed to build learning health systems were also recurrent themes.

Open Science and eGEMs: Our Role in Supporting a Culture of Collaboration in Learning Health Systems.

"Open science" includes a variety of approaches to facilitate greater access to data and the information produced by processes of scientific inquiry. Recently, the health sciences community has been grappling with the issue of potential pathways and models to achieve the goals of open science-namely, to create and rapidly share reproducible health research. eGEMs' continued dedication to and milestones regarding the publication of innovative, useful, and timely research to help contribute to the push towards open science is discussed, as well as the EDM Forum's new data sharing platform, CIELO. Although strides have been made, there is still more work to be done to help health sciences community truly embrace open science.

Incorporating Patient-Reported Outcomes Into Health Care To Engage Patients And Enhance Care.

The provision of patient-centered care requires a health care environment that fosters engagement between patients and their health care team. One way to encourage patient-centered care is to incorporate patient-reported outcomes into clinical settings. Collecting these outcomes in routine care ensures that important information only the patient can provide is captured. This provides insights into patients' experiences of symptoms, quality of life, and functioning; values and preferences; and goals for health care. Previously embraced in the research realm, patient-reported outcomes have started to play a role in successful shared decision making, which can enhance the safe and effective delivery of health care. We examine the opportunities for using patient-reported outcomes to enhance care delivery and outcomes as health care information needs and technology platforms change. We highlight emerging practices in which patient-reported outcomes provide value to patients and clinicians and improve care delivery. Finally, we examine present and future challenges to maximizing the use of patient-reported outcomes in the clinic.

Developing evidence that is fit for purpose: a framework for payer and research dialogue.

OBJECTIVES: Matching the supply and demand of evidence requires an understanding of when more evidence is needed, as well as the type of evidence that will meet this need. This article describes efforts to develop and refine a decision-making framework that considers payers' perspectives on the utility of evidence generated by different types of research methods, including real-world evidence. STUDY DESIGN: Conceptual framework development with subsequent testing during a roundtable dialogue. METHODS: The framework development process included a literature scan to identify existing frameworks and relevant articles on payer decision making. The framework was refined during a stand-alone roundtable in December 2013 hosted by the research team, which included representatives from public and private payers, pharmacy benefit management, the life sciences industry, and researchers. The roundtable discussion also included an application of the framework to 3 case studies. RESULTS: Application of the framework to the clinical scenarios and the resulting discussion provided key insights into when new evidence is needed to inform payer decision making and what questions should be addressed. Payers are not necessarily seeking more evidence about treatment efficacy; rather, they are seeking more evidence for relevant end points that illustrate the differences between treatment alternatives that can justify the resources required to change practice. In addition, payers are interested in obtaining new evidence that goes beyond efficacy, with an emphasis on effectiveness, longer-term safety, and delivery system impact. CONCLUSIONS: We believe that our decision-making framework is a useful tool to increase dialogue between evidence generators and payers, while also allowing for greater efficiency in the research process.

Infrastructure to support learning health systems: are we there yet? Innovative solutions and lessons learned from American Recovery and Reinvestment Act CER investments.

The 11 big health data networks participating in the AcademyHealth Electronic Data Methods Forum represent cutting-edge efforts to harness the power of big health data for research and quality improvement. This paper is a comparative case study based on site visits conducted with a subset of these large infrastructure grants funded through the Recovery Act, in which four key issues emerge that can inform the evolution of learning health systems, including the importance of acknowledging the challenges of scaling specialized expertise needed to manage and run CER networks; the delicate balance between privacy protections and the utility of distributed networks; emerging community engagement strategies; and the complexities of developing a robust business model for multi-use networks.

American Recovery and Reinvestment Act-comparative effectiveness research infrastructure investments: emerging data resources, tools and publications.

The Recovery Act provided a substantial, one-time investment in data infrastructure for comparative effectiveness research (CER). A review of the publications, data, and tools developed as a result of this support has informed understanding of the level of effort undertaken by these projects. Structured search queries, as well as outreach efforts, were conducted to identify and review resources from American Recovery and Reinvestment Act of 2009 CER projects building electronic clinical data infrastructure. The findings from this study provide a spectrum of productivity across a range of topics and settings. A total of 451 manuscripts published in 192 journals, and 141 data resources and tools were identified and address gaps in evidence on priority populations, conditions, and the infrastructure needed to support CER.

eGEMs' Early Adventures in Open Access Publishing.

INTRODUCTION: In January 2013 AcademyHealth officially launched eGEMs (Generating Evidence and Methods to improve patient outcomes) to rapidly disseminate peer-reviewed approaches using electronic health data (EHD) to advance research and quality improvement (QI), with the overall goal of improving patient and community outcomes. Inspired by the publication of eGEMs 50(th) paper, Dr. Erin Holve, eGEMs editor-in-chief reviews the EDM Forum's early experiences with open access publishing. EGEMS TO DATE: As of the end of September 2014 eGEMs has published 59 manuscripts and received nearly 150 submissions. These early findings demonstrate eGEMs is filling a need for dissemination outlets that bridge the gap between the health research and practice communities. Published papers are distributed across the EDM Forum's four thematic domains: governance (n=5), informatics (n=14), methods (n=13) and learning health systems (n=27). While system design issues are a consistent theme, papers addressing priority health topics such as diabetes, asthma, and obesity are frequently submitted. Authors include more than two hundred experts in the field representing nearly all of the EDM Forum's core stakeholder groups: research/QI, nonprofit/policy, healthcare delivery, government, industry, and patients/consumers. WHAT'S NEXT: With the help of our diverse community, eGEMs will continue to expand its depth and reach. Forthcoming special issues on community-level transformation using health IT, and ways to improve user-experience and system design will add to the journal's robust portfolio of work identifying and addressing shared challenges using EHD. The EDM Forum, working closely with our partners at the Agency for Healthcare Research and Quality, will work diligently to ensure eGEMs is accelerating the pace at which the community translates and disseminates key lessons, with the ultimate goal of helping transform knowledge into actions that can improve health and health care.

Sustaining the effective use of health care data: a message from the editors.

INTRODUCTION: Over the past decade, several initiatives have funded large projects to develop clinical research data infrastructures totaling several hundred million dollars. While most of this funding has ended or is expected to end soon, the projects themselves must struggle to continue operations beyond the initial funding. Examples of sustained research-data infrastructures are lacking, and recommended approaches to improve sustainability of developing infrastructures are even rarer. Early on, the Electronic Data Methods (EDM) Forum-and the Agency for Healthcare Research and Quality (AHRQ) as its sponsor-recognized the need to study strategies for sustainability. THEMES: Three prominent themes relating to sustainability arise among the articles in this special issue: the importance of project maturity, commercialization activities, and stakeholder support. Maturity was relevant to all the papers since a project's maturity directly influences the opportunities that are available, while commercialization and stakeholder support emerged from comparisons among subsets of articles. NEXT STEPS: The papers in this issue create a useful initial set of case studies to help in understanding sustainability issues for data infrastructures needed for research and QI. Each paper includes important lessons learned from the authors' experience with the different projects that should resonate with the broader fields of clinical research and clinical research informatics. There is an ongoing need for greater understanding of sustainability beyond what this issue provides. As more case studies of sustainability are accumulated, it is expected even more important themes will emerge from qualitative reviews that can eventually be demonstrated quantitatively.

A comprehensive framework for data quality assessment in CER.

The panel addresses the urgent need to ensure that comparative effectiveness research (CER) findings derived from diverse and distributed data sources are based on credible, high-quality data; and that the methods used to assess and report data quality are consistent, comprehensive, and available to data consumers. The panel consists of representatives from four teams leveraging electronic clinical data for CER, patient centered outcomes research (PCOR), and quality improvement (QI) and seeks to change the current paradigm where data quality assessment (DQA) is performed "behind the scenes" using one-off project specific methods. The panelists will present their process of harmonizing existing models for describing and measuring clinical data quality and will describe a comprehensive integrated framework for assessing and reporting DQA findings. The collaborative project is supported by the Electronic Data Methods (EDM) Forum, a three-year grant from the Agency for Healthcare Research and Quality (AHRQ) to facilitate learning and foster collaboration across a set of CER, PCOR, and QI projects designed to build infrastructure and methods for collecting and analyzing prospective data from electronic clinical data .

Lessons from the Electronic Data Methods Forum: collaboration at the frontier of comparative effectiveness research, patient-centered outcomes research, and quality improvement.

BACKGROUND: The Electronic Data Methods (EDM) Forum, with support from the Agency for Healthcare Research and Quality, exists to advance knowledge and practice on the use of electronic clinical data (ECD) for comparative effectiveness research, patient-centered outcomes research, and quality improvement (QI). The EDM Forum facilitates collaboration between the Prospective Outcome Systems using Patient-specific Electronic data to Compare Tests and therapies, Scalable Distributed Research Network, and Enhanced registry projects funded by Agency for Healthcare Research and Quality. OBJECTIVES: This overview describes a second set of papers commissioned by the EDM Forum, published in this supplement to Medical Care. The papers that are included discuss challenges and innovations from the research and QI community using ECD. CONCLUSIONS: The papers in this supplement provide lessons learned based on experiences building transparent, scalable, reusable networks for research and QI. Through these papers, and a new open access e-journal, eGEMs, the EDM Forum is working to advance the science of health research and QI using ECD to improve patient outcomes.

eGEMs: Pragmatic Publishing to Build a Learning Health System.

The Electronic Data Methods (EDM) Forum announces the launch of eGEMs (Generating Evidence and Methods to improve patient outcomes), a new, free, open access, peer-reviewed e-publication. eGEMs aims to disseminate innovative ideas about how electronic clinical data (ECD) can be leveraged in comparative effectiveness research (CER), patient-centered outcomes research (PCOR) and quality improvement (QI). eGEMs seeks submissions in four main topic areas: analytic methods, clinical informatics, governance, and the learning health system. The goal of eGEMs is ultimately to promote dialogue and the sharing of ideas between researchers and other stakeholders in a credible and timely way, extending the efforts of the EDM Forum.

Ensuring Support for Research and Quality Improvement (QI) Networks: Four Pillars of Sustainability-An Emerging Framework.

Multi-institutional research and quality improvement (QI) projects using electronic clinical data (ECD) hold great promise for improving quality of care and patient outcomes but typically require significant infrastructure investments both to initiate and maintain the project over its duration. Consequently, it is important for these projects to think holistically about sustainability to ensure their long-term success. Four "pillars" of sustainability are discussed based on the experiences of EDM Forum grantees and other research and QI networks. These include trust and value, governance, management, and financial and administrative support. Two "foundational considerations," adaptive capacity and policy levers, are also discussed.

Opportunities and challenges for comparative effectiveness research (CER) with Electronic Clinical Data: a perspective from the EDM forum.

INTRODUCTION: The Electronic Data Methods (EDM) Forum brings together perspectives from the Prospective Outcome Systems using Patient-specific Electronic data to Compare Tests and therapies (PROSPECT) studies, the Scalable Distributed Research Networks, and the Enhanced Registries projects. This paper discusses challenges faced by the research teams as part of their efforts to develop electronic clinical data (ECD) infrastructure to support comparative effectiveness research (CER). The findings reflect a set of opportunities for transdisciplinary learning, and will ideally enhance the transparency and generalizability of CER using ECD. METHODS: Findings are based on 6 exploratory site visits conducted under naturalistic inquiry in the spring of 2011. Themes, challenges, and innovations were identified in the visit summaries through coding, keyword searches, and review for complex concepts. RESULTS: : The identified overarching challenges and emerging opportunities include: the substantial level of effort to establish and sustain data sharing partnerships; the importance of understanding the strengths and limitations of clinical informatics tools, platforms, and models that have emerged to enable research with ECD; the need for rigorous methods to assess data validity, quality, and context for multisite studies; and, emerging opportunities to achieve meaningful patient and consumer engagement and work collaboratively with multidisciplinary teams. DISCUSSION: The new infrastructure must evolve to serve a diverse set of potential users and must scale to address a range of CER or patient-centered outcomes research (PCOR) questions. To achieve this aim-to improve the quality, transparency, and reproducibility of CER and PCOR-a high level of collaboration and support is necessary to foster partnership and best practices as part of the EDM Forum.