Enabling Open Science for Health Research: Collaborative Informatics Environment for Learning on Health Outcomes (CIELO).

BACKGROUND: There is an emergent and intensive dialogue in the United States with regard to the accessibility, reproducibility, and rigor of health research. This discussion is also closely aligned with the need to identify sustainable ways to expand the national research enterprise and to generate actionable results that can be applied to improve the nation's health. The principles and practices of Open Science offer a promising path to address both goals by facilitating (1) increased transparency of data and methods, which promotes research reproducibility and rigor; and (2) cumulative efficiencies wherein research tools and the output of research are combined to accelerate the delivery of new knowledge in proximal domains, thereby resulting in greater productivity and a reduction in redundant research investments. OBJECTIVES: AcademyHealth's Electronic Data Methods (EDM) Forum implemented a proof-of-concept open science platform for health research called the Collaborative Informatics Environment for Learning on Health Outcomes (CIELO). METHODS: The EDM Forum conducted a user-centered design process to elucidate important and high-level requirements for creating and sustaining an open science paradigm. RESULTS: By implementing CIELO and engaging a variety of potential users in its public beta testing, the EDM Forum has been able to elucidate a broad range of stakeholder needs and requirements related to the use of an open science platform focused on health research in a variety of "real world" settings. CONCLUSIONS: Our initial design and development experience over the course of the CIELO project has provided the basis for a vigorous dialogue between stakeholder community members regarding the capabilities that will add the greatest value to an open science platform for the health research community. A number of important questions around user incentives, sustainability, and scalability will require further community dialogue and agreement.

Distributed Data Networks That Support Public Health Information Needs.

INTRODUCTION: Data networks, consisting of pooled electronic health data assets from health care providers serving different patient populations, promote data sharing, population and disease monitoring, and methods to assess interventions. Better understanding of data networks, and their capacity to support public health objectives, will help foster partnerships, expand resources, and grow learning health systems. METHODS: We conducted semistructured interviews with 16 key informants across the United States, identified as network stakeholders based on their respective experience in advancing health information technology and network functionality. Key informants were asked about their experience with and infrastructure used to develop data networks, including each network's utility to identify and characterize populations, usage, and sustainability. RESULTS: Among 11 identified data networks representing hundreds of thousands of patients, key informants described aggregated health care clinical data contributing to population health measures. Key informant interview responses were thematically grouped to illustrate how networks support public health, including (1) infrastructure and information sharing; (2) population health measures; and (3) network sustainability. CONCLUSION: Collaboration between clinical data networks and public health entities presents an opportunity to leverage infrastructure investments to support public health. Data networks can provide resources to enhance population health information and infrastructure.

Lessons from the Electronic Data Methods Forum: collaboration at the frontier of comparative effectiveness research, patient-centered outcomes research, and quality improvement.

BACKGROUND: The Electronic Data Methods (EDM) Forum, with support from the Agency for Healthcare Research and Quality, exists to advance knowledge and practice on the use of electronic clinical data (ECD) for comparative effectiveness research, patient-centered outcomes research, and quality improvement (QI). The EDM Forum facilitates collaboration between the Prospective Outcome Systems using Patient-specific Electronic data to Compare Tests and therapies, Scalable Distributed Research Network, and Enhanced registry projects funded by Agency for Healthcare Research and Quality. OBJECTIVES: This overview describes a second set of papers commissioned by the EDM Forum, published in this supplement to Medical Care. The papers that are included discuss challenges and innovations from the research and QI community using ECD. CONCLUSIONS: The papers in this supplement provide lessons learned based on experiences building transparent, scalable, reusable networks for research and QI. Through these papers, and a new open access e-journal, eGEMs, the EDM Forum is working to advance the science of health research and QI using ECD to improve patient outcomes.

Opportunities and challenges for comparative effectiveness research (CER) with Electronic Clinical Data: a perspective from the EDM forum.

INTRODUCTION: The Electronic Data Methods (EDM) Forum brings together perspectives from the Prospective Outcome Systems using Patient-specific Electronic data to Compare Tests and therapies (PROSPECT) studies, the Scalable Distributed Research Networks, and the Enhanced Registries projects. This paper discusses challenges faced by the research teams as part of their efforts to develop electronic clinical data (ECD) infrastructure to support comparative effectiveness research (CER). The findings reflect a set of opportunities for transdisciplinary learning, and will ideally enhance the transparency and generalizability of CER using ECD. METHODS: Findings are based on 6 exploratory site visits conducted under naturalistic inquiry in the spring of 2011. Themes, challenges, and innovations were identified in the visit summaries through coding, keyword searches, and review for complex concepts. RESULTS: : The identified overarching challenges and emerging opportunities include: the substantial level of effort to establish and sustain data sharing partnerships; the importance of understanding the strengths and limitations of clinical informatics tools, platforms, and models that have emerged to enable research with ECD; the need for rigorous methods to assess data validity, quality, and context for multisite studies; and, emerging opportunities to achieve meaningful patient and consumer engagement and work collaboratively with multidisciplinary teams. DISCUSSION: The new infrastructure must evolve to serve a diverse set of potential users and must scale to address a range of CER or patient-centered outcomes research (PCOR) questions. To achieve this aim-to improve the quality, transparency, and reproducibility of CER and PCOR-a high level of collaboration and support is necessary to foster partnership and best practices as part of the EDM Forum.

Building the informatics infrastructure for comparative effectiveness research (CER): a review of the literature.

BACKGROUND: Technological advances in clinical informatics have made large amounts of data accessible and potentially useful for research. As a result, a burgeoning literature addresses efforts to bridge the fields of health services research and biomedical informatics. The Electronic Data Methods Forum review examines peer-reviewed literature at the intersection of comparative effectiveness research and clinical informatics. The authors are specifically interested in characterizing this literature and identifying cross-cutting themes and gaps in the literature. METHODS: A 3-step systematic literature search was conducted, including a structured search of PubMed, manual reviews of articles from selected publication lists, and manual reviews of research activities based on prospective electronic clinical data. Two thousand four hundred thirty-five citations were identified as potentially relevant. Ultimately, a full-text review was performed for 147 peer-reviewed papers. RESULTS: One hundred thirty-two articles were selected for inclusion in the review. Of these, 88 articles are the focus of the discussion in this paper. Three types of articles were identified, including papers that: (1) provide historical context or frameworks for using clinical informatics for research, (2) describe platforms and projects, and (3) discuss issues, challenges, and applications of natural language processing. In addition, 2 cross-cutting themes emerged: the challenges of conducting research in the absence of standardized ontologies and data collection; and unique data governance concerns related to the transfer, storage, deidentification, and access to electronic clinical data. Finally, the authors identified several current gaps on important topics such as the use of clinical informatics for cohort identification, cloud computing, and single point access to research data.

The Electronic Data Methods (EDM) forum for comparative effectiveness research (CER).

BACKGROUND: AcademyHealth convened the Electronic Data Methods (EDM) Forum to collect, synthesize, and share lessons from eleven projects that are building infrastructure and using electronic clinical data for comparative effectiveness research (CER) and patient-centered outcomes research (PCOR). This paper provides a brief review of participating projects and provides a framework of common challenges. METHODS: EDM Forum staff conducted a text review of relevant grant programs' funding opportunity announcements; projects' research plans; and available information on projects' websites. Additional information was obtained from presentations provided by each project; phone calls with project principal investigators, affiliated partners, and staff from the Agency for Healthcare Research and Quality (AHRQ); and six site visits. RESULTS: Projects participating in the EDM Forum are building infrastructure and developing innovative strategies to address a set of methodological, and data and informatics challenges, here identified in a common framework. The eleven networks represent more than 20 states and include a range of partnership models. Projects vary substantially in size, from 11,000 to more than 7.5 million individuals. Nearly all of the AHRQ priority populations and conditions are addressed. DISCUSSION: In partnership with the projects, the EDM Forum is focused on identifying and sharing lessons learned to advance the national dialogue on the use of electronic clinical data to conduct CER and PCOR. These efforts have the shared goal of addressing challenges in traditional research studies and data sources, and aim to build infrastructure and generate evidence to support a learning health care system that can improve patient outcomes.

Health services research doctoral core competencies.

This manuscript presents an initial description of doctoral level core competencies for health services research (HSR). The competencies were developed by a review of the literature, text analysis of institutional accreditation self-studies submitted to the Council on Education for Public Health, and a consensus conference of HSR educators from US educational institutions. The competencies are described in broad terms which reflect the unique expertise, interests, and preferred learning methods of academic HSR programs. This initial set of core competencies is published to generate further dialogue within and outside of the US about the most important learning objectives and methods for HSR training and to clarify the unique skills of HSR training program graduates.

A Framework for Classification of Electronic Health Data Extraction-Transformation-Loading Challenges in Data Network Participation.

BACKGROUND: Contributing health data to national, regional, and local networks or registries requires data stored in local systems with local structures and codes to be extracted, transformed, and loaded into a standard format called a Common Data Model (CDM). These processes called Extract, Transform, Load (ETL) require data partners or contributors to invest in costly technical resources with specialized skills in data models, terminologies, and programming. Given the wide range of tasks, skills, and technologies required to transform data into a CDM, a classification of ETL challenges can help identify needed resources, which in turn may encourage data partners with less-technical capabilities to participate in data-sharing networks. METHODS: We conducted key-informant interviews with data partner representatives to survey the ETL challenges faced in clinical data research networks (CDRNs) and registries. A list of ETL challenges, organized into six themes was vetted during a one-day workshop with a wide range of network stakeholders including data partners, researchers, and policy experts. RESULTS: We identified 24 technical ETL challenges related to the data sharing process. All of these ETL challenges were rated as "important" or "very important" by workshop participants using a five point Likert scale. Based on these findings, a framework for categorizing ETL challenges according to ETL phases, themes, and levels of data network participation was developed. CONCLUSIONS: Overcoming ETL technical challenges require significant investments in a broad array of information technologies and human resources. Identifying these technical obstacles can inform optimal resource allocation to minimize the barriers and cost of entry for new data partners into extant networks, which in turn can expand data networks' inclusiveness and diversity. This paper offers pertinent information and guiding framework that are relevant for data partners in ascertaining challenges associated with contributing data in data networks.

AcademyHealth's Delivery System Science Fellowship: Training Embedded Researchers to Design, Implement, and Evaluate New Models of Care.

AcademyHealth's Delivery System Science Fellowship (DSSF) provides a paid postdoctoral pragmatic learning experience to build capacity within learning healthcare systems to conduct research in applied settings. The fellowship provides hands-on training and professional leadership opportunities for researchers. Since its inception in 2012, the program has grown rapidly, with 16 health systems participating in the DSSF to date. In addition to specific projects conducted within health systems (and numerous publications associated with those initiatives), the DSSF has made several broader contributions to the field, including defining delivery system science, identifying a set of training objectives for researchers working in delivery systems, and developing a national collaborative network of care delivery organizations, operational leaders, and trainees. The DSSF is one promising approach to support higher-value care by promoting continuous learning and improvement in health systems.

Framework To Guide The Collection And Use Of Patient-Reported Outcome Measures In The Learning Healthcare System.

BACKGROUND: Web-based collection of patient-reported outcome measures (PROMs) in clinical practice is expanding rapidly as electronic health records include web portals for patients to report standardized assessments of their symptoms. As the value of PROMs in patient care expands, a framework to guide the implementation planning, collection, and use of PROs to serve multiple goals and stakeholders is needed. METHODS: We identified diverse clinical, quality, and research settings where PROMs have been successfully integrated into care and routinely collected and analyzed drivers of successful implementation. Findings are based on key informant interviews with 46 individuals representing 38 organizations, of whom 40 participated in a webinars series, and 25 attended an in-person workshop designed to enable broad stakeholder input, review and refinement of the proposed PROMs implementation model. Stakeholders identified differing uses of PROMs to support: 1) individual patient care decisions, 2) quality improvement initiatives, 3) payer mandates, and 4) population health and research. RESULTS: The implementation framework and steps that are consistently identified by stakeholders as best practices to guide PROM capture and use are described. Of note, participants indicate that web-based informatics tools are necessary but not sufficient for PROM use, suggesting that successful PROM implementation requires integration into clinic operations and careful planning for user's analytic needs. Each of the four identified uses may require implementation modifications at each step to assure optimal use. CONCLUSIONS: The proposed framework will guide future PROM implementation efforts across learning health care systems to assure that complete PROMs are captured at the correct time, and with associated risk factors, to generate meaningful information to serve diverse stakeholders.

Open Science and eGEMs: Our Role in Supporting a Culture of Collaboration in Learning Health Systems.

"Open science" includes a variety of approaches to facilitate greater access to data and the information produced by processes of scientific inquiry. Recently, the health sciences community has been grappling with the issue of potential pathways and models to achieve the goals of open science-namely, to create and rapidly share reproducible health research. eGEMs' continued dedication to and milestones regarding the publication of innovative, useful, and timely research to help contribute to the push towards open science is discussed, as well as the EDM Forum's new data sharing platform, CIELO. Although strides have been made, there is still more work to be done to help health sciences community truly embrace open science.

Concordium 2015: Strategic Uses of Evidence to Transform Delivery Systems.

In September 2015 the EDM Forum hosted AcademyHealth's newest national conference, Concordium. The 11 papers featured in the eGEMs "Concordium 2015" special issue successfully reflect the major themes and issues discussed at the meeting. Many of the papers address informatics or methodological approaches to natural language processing (NLP) or text analysis, which is indicative of the importance of analyzing text data to gain insights into care coordination and patient-centered outcomes. Perspectives on the tools and infrastructure requirements that are needed to build learning health systems were also recurrent themes.

Incorporating Patient-Reported Outcomes Into Health Care To Engage Patients And Enhance Care.

The provision of patient-centered care requires a health care environment that fosters engagement between patients and their health care team. One way to encourage patient-centered care is to incorporate patient-reported outcomes into clinical settings. Collecting these outcomes in routine care ensures that important information only the patient can provide is captured. This provides insights into patients' experiences of symptoms, quality of life, and functioning; values and preferences; and goals for health care. Previously embraced in the research realm, patient-reported outcomes have started to play a role in successful shared decision making, which can enhance the safe and effective delivery of health care. We examine the opportunities for using patient-reported outcomes to enhance care delivery and outcomes as health care information needs and technology platforms change. We highlight emerging practices in which patient-reported outcomes provide value to patients and clinicians and improve care delivery. Finally, we examine present and future challenges to maximizing the use of patient-reported outcomes in the clinic.

A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data.

OBJECTIVE: Harmonized data quality (DQ) assessment terms, methods, and reporting practices can establish a common understanding of the strengths and limitations of electronic health record (EHR) data for operational analytics, quality improvement, and research. Existing published DQ terms were harmonized to a comprehensive unified terminology with definitions and examples and organized into a conceptual framework to support a common approach to defining whether EHR data is 'fit' for specific uses. MATERIALS AND METHODS: DQ publications, informatics and analytics experts, managers of established DQ programs, and operational manuals from several mature EHR-based research networks were reviewed to identify potential DQ terms and categories. Two face-to-face stakeholder meetings were used to vet an initial set of DQ terms and definitions that were grouped into an overall conceptual framework. Feedback received from data producers and users was used to construct a draft set of harmonized DQ terms and categories. Multiple rounds of iterative refinement resulted in a set of terms and organizing framework consisting of DQ categories, subcategories, terms, definitions, and examples. The harmonized terminology and logical framework's inclusiveness was evaluated against ten published DQ terminologies. RESULTS: Existing DQ terms were harmonized and organized into a framework by defining three DQ categories: (1) Conformance (2) Completeness and (3) Plausibility and two DQ assessment contexts: (1) Verification and (2) Validation. Conformance and Plausibility categories were further divided into subcategories. Each category and subcategory was defined with respect to whether the data may be verified with organizational data, or validated against an accepted gold standard, depending on proposed context and uses. The coverage of the harmonized DQ terminology was validated by successfully aligning to multiple published DQ terminologies. DISCUSSION: Existing DQ concepts, community input, and expert review informed the development of a distinct set of terms, organized into categories and subcategories. The resulting DQ terms successfully encompassed a wide range of disparate DQ terminologies. Operational definitions were developed to provide guidance for implementing DQ assessment procedures. The resulting structure is an inclusive DQ framework for standardizing DQ assessment and reporting. While our analysis focused on the DQ issues often found in EHR data, the new terminology may be applicable to a wide range of electronic health data such as administrative, research, and patient-reported data. CONCLUSION: A consistent, common DQ terminology, organized into a logical framework, is an initial step in enabling data owners and users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary. Future work will leverage the framework and terminology to develop reusable data quality assessment and reporting methods.

Developing evidence that is fit for purpose: a framework for payer and research dialogue.

OBJECTIVES: Matching the supply and demand of evidence requires an understanding of when more evidence is needed, as well as the type of evidence that will meet this need. This article describes efforts to develop and refine a decision-making framework that considers payers' perspectives on the utility of evidence generated by different types of research methods, including real-world evidence. STUDY DESIGN: Conceptual framework development with subsequent testing during a roundtable dialogue. METHODS: The framework development process included a literature scan to identify existing frameworks and relevant articles on payer decision making. The framework was refined during a stand-alone roundtable in December 2013 hosted by the research team, which included representatives from public and private payers, pharmacy benefit management, the life sciences industry, and researchers. The roundtable discussion also included an application of the framework to 3 case studies. RESULTS: Application of the framework to the clinical scenarios and the resulting discussion provided key insights into when new evidence is needed to inform payer decision making and what questions should be addressed. Payers are not necessarily seeking more evidence about treatment efficacy; rather, they are seeking more evidence for relevant end points that illustrate the differences between treatment alternatives that can justify the resources required to change practice. In addition, payers are interested in obtaining new evidence that goes beyond efficacy, with an emphasis on effectiveness, longer-term safety, and delivery system impact. CONCLUSIONS: We believe that our decision-making framework is a useful tool to increase dialogue between evidence generators and payers, while also allowing for greater efficiency in the research process.

Health benefits in 2003: premiums reach thirteen-year high as employers adopt new forms of cost sharing.

This paper reports changes in job-based health insurance from spring 2002 to spring 2003. The cost of health insurance rose 13.9 percent, the highest rate of increase since 1990. Employers required larger contributions from employees for the monthly cost of health insurance. Separate copayments and deductibles for hospital services have become commonplace, and provider networks have broadened. There was no change in the percentage of employers offering health plans to their workers. Employers indicate little confidence in any future strategies for controlling health care costs.

Job-based health benefits in 2002: some important trends.

Based on a national survey of 2,014 randomly selected public and private firms with three or more workers, this paper reports changes in employer-based health insurance from spring 2001 to spring 2002. The cost of health insurance rose 12.7 percent, the highest rate of growth since 1990. Employee contributions for health insurance rose in 2002, from $30 to $38 for single coverage and from $150 to $174 for family coverage. Deductibles and copayments rose also, and employers adopted formularies and three-tier cost-sharing formulas to control prescription drug expenses. PPO and HMO enrollment rose, while the percentage of small employers offering health benefits fell. Because increasing claims expenses rather than the underwriting cycle are the major driver of rising premiums, double-digit growth appears likely to continue.

Health benefits in 2004: four years of double-digit premium increases take their toll on coverage.

This paper reports changes in employer-based insurance during the past year and since 2001. From spring 2003 to spring 2004, premiums increased 11.2 percent (compared with 13.9 percent last year). Since 2000, premiums have increased 59 percent. Since 2001, employee contributions have grown by 57 percent for single coverage and 49 percent for family coverage, and the percentage of workers covered by their own employer's health plan has fallen from 65 percent in 2001 to 61 percent in 2004. The worst of the current round of premium inflation appears to be over, but employers plan to increase employee cost sharing next year [corrected]

Small business executives and health insurance: findings from a national survey of very small firms.

Previous researchers have documented that very small businesses (3-24 workers) are less likely to offer employees health insurance than larger corporations. This study supplements previous findings on the prevalence of health insurance among small firms. The authors also attempt to illuminate reasons behind coverage decisions by interviewing small business owners and executives, who most often make health benefits decisions on behalf of their employees. The study examines attitudes about health insurance, opinions, and practices in these very small firms, and the response of small business owners to policy alternatives designed to expand coverage in small businesses.