Applying Machine Learning Techniques to Implementation Science.

Machine learning (ML) approaches could expand the usefulness and application of implementation science methods in clinical medicine and public health settings. The aim of this viewpoint is to introduce a roadmap for applying ML techniques to address implementation science questions, such as predicting what will work best, for whom, under what circumstances, and with what predicted level of support, and what and when adaptation or deimplementation are needed. We describe how ML approaches could be used and discuss challenges that implementation scientists and methodologists will need to consider when using ML throughout the stages of implementation.

Integrating a focus on health equity in implementation science: Case examples from the national cancer institute's implementation science in cancer control centers (ISC3) network.

Background: A Health Equity Task Force (HETF) of members from seven Centers funded by the National Cancer Institute's (NCI) Implementation Science in Cancer Control Centers (ISC3) network sought to identify case examples of how Centers were applying a focus on health equity in implementation science to inform future research and capacity-building efforts. Methods: HETF members at each ISC3 collected information on how health equity was conceptualized, operationalized, and addressed in initial research and capacity-building efforts across the seven ISC3 Centers funded in 2019-2020. Each Center completed a questionnaire assessing five health equity domains central to implementation science (e.g., community engagement; implementation science theories, models, and frameworks (TMFs); and engaging underrepresented scholars). Data generated illustrative examples from these five domains. Results: Centers reported a range of approaches focusing on health equity in implementation research and capacity-building efforts, including (1) engaging diverse community partners/settings in making decisions about research priorities and projects; (2) applying health equity within a single TMF applied across projects or various TMFs used in specific projects; (3) evaluating health equity in operationalizing and measuring health and implementation outcomes; (4) building capacity for health equity-focused implementation science among trainees, early career scholars, and partnering organizations; and (5) leveraging varying levels of institutional resources and efforts to engage, include, and support underrepresented scholars. Conclusions: Examples of approaches to integrating health equity across the ISC3 network can inform other investigators and centers' efforts to build capacity and infrastructure to support growth and expansion of health equity-focused implementation science.

Participatory logic modeling in a multi-site initiative to advance implementation science.

BACKGROUND: Logic models map the short-term and long-term outcomes that are expected to occur with a program, and thus are an essential tool for evaluation. Funding agencies, especially in the United States (US), have encouraged the use of logic models among their grantees. They also use logic models to clarify expectations for their own funding initiatives. It is increasingly recognized that logic models should be developed through a participatory approach which allows input from those who carry out the program being evaluated. While there are many positive examples of participatory logic modeling, funders have generally not engaged grantees in developing the logic model associated with their own initiatives. This article describes an instance where a US funder of a multi-site initiative fully engaged the funded organizations in developing the initiative logic model. The focus of the case study is Implementation Science Centers in Cancer Control (ISC3), a multi-year initiative funded by the National Cancer Institute. METHODS: The reflective case study was collectively constructed by representatives of the seven centers funded under ISC3. Members of the Cross-Center Evaluation (CCE) Work Group jointly articulated the process through which the logic model was developed and refined. Individual Work Group members contributed descriptions of how their respective centers reviewed and used the logic model. Cross-cutting themes and lessons emerged through CCE Work Group meetings and the writing process. RESULTS: The initial logic model for ISC3 changed in significant ways as a result of the input of the funded groups. Authentic participation in the development of the logic model led to strong buy-in among the centers, as evidenced by their utilization. The centers shifted both their evaluation design and their programmatic strategy to better accommodate the expectations reflected in the initiative logic model. CONCLUSIONS: The ISC3 case study demonstrates how participatory logic modeling can be mutually beneficial to funders, grantees and evaluators of multi-site initiatives. Funded groups have important insights about what is feasible and what will be required to achieve the initiative's stated objectives. They can also help identify the contextual factors that either inhibit or facilitate success, which can then be incorporated into both the logic model and the evaluation design. In addition, when grantees co-develop the logic model, they have a better understanding and appreciation of the funder's expectations and thus are better positioned to meet those expectations.

Participatory Logic Modeling in a Multi-Site Initiative to Advance Implementation Science.

Background: It is increasingly being recognized that logic models should be developed through a participatory approach which allows input from those who carry out the program being evaluated. While there are many positive examples of participatory logic modeling, funders have generally not used this approach in the context of multi-site initiatives. This article describes an instance where the funder and evaluator of a multi-site initiative fully engaged the funded organizations in developing the initiative logic model. The focus of the case study is Implementation Science Centers in Cancer Control (ISC 3 ), a multi-year initiative funded by the National Cancer Institute (NCI). Methods: The case study was collectively constructed by representatives of the seven centers funded under ISC 3 . Members of the Cross-Center Evaluation (CCE) Work Group jointly articulated the process through which the logic model was developed and refined. Individual Work Group members contributed descriptions of how their respective centers reviewed and used the logic model. Cross-cutting themes and lessons emerged through CCE Work Group meetings and the writing process. Results: The initial logic model for ISC 3 changed in significant ways as a result of the input of the funded groups. Authentic participation in the development of the logic model led to strong buy-in among the centers, as evidenced by their utilization. The centers shifted both their evaluation design and their programmatic strategy to better accommodate the expectations reflected in the initiative logic model. Conclusions: The ISC 3 case study provides a positive example of how participatory logic modeling can be mutually beneficial to funders, grantees and evaluators of multi-site initiatives. Funded groups have important insights about what is feasible and what will be required to achieve the initiative's stated objectives. They can also help identify the contextual factors that either inhibit or facilitate success, which can then be incorporated into both the logic model and the evaluation design. In addition, when grantees co-develop the logic model, they have a better understanding and appreciation of the funder's expectations, and thus are better positioned to meet those expectations.

Precision Ecologic Medicine: Tailoring Care to Mitigate Impacts of Climate Change.

As recent extreme weather events demonstrate, climate change presents unprecedented and increasing health risks, disproportionately so for disadvantaged communities in the U.S. already experiencing health disparities. As patients in these frontline communities live through extreme weather events, socioeconomic and health stressors are compounded; thus, their healthcare teams will need tools to provide precision ecologic medicine approaches to their care. Many primary care teams are taking actionable steps to bring community-level socioeconomic data ("community vital signs") into electronic medical records, to facilitate tailoring care based on a given patient's circumstances. This work can be extended to include environmental risk data, thus equipping healthcare teams with an awareness of clinical and community vital signs and making them better positioned to mitigate climate impacts on health. For example, if healthcare teams can easily identify patients who have multiple chronic conditions and live in an urban heat island, they can proactively arrange to "prescribe" an air conditioner, heat pump, and/or air purifier. Or, when a severe storm/heat event/poor air quality event is predicted, they can take preemptive steps to get help to patients at high medical and socioeconomic risk, rather than waiting for them to arrive in the emergency department. Advances in health information technologies now make it technically feasible to integrate a wealth of publicly-available community-level data into EMRs. Efforts to bring this contextual data into clinical settings must be accelerated to equip healthcare teams to provide precision ecologic medicine interventions to their patients.

A Practice-Based Research Network (PBRN) Roadmap for Evaluating COVID-19 in Community Health Centers: A Report From the OCHIN PBRN.

BACKGROUND: Primary care practice-based research networks (PBRNs) are critical laboratories for generating evidence from real-world settings, including studying natural experiments. Primary care's response to the novel coronavirus-19 (COVID-19) pandemic is arguably the most impactful natural experiment in our lifetime. EVALUATING THE IMPACT OF COVID-19: We briefly describe the OCHIN PBRN of community health centers (CHCs), its partnership with implementation scientists, and how we are leveraging this infrastructure and expertise to create a rapid research response evaluating how CHCs across the country responded to the COVID-19 pandemic. COVID-19 RESEARCH ROADMAP: Our research agenda focuses on asking: How has care delivery in CHCs changed due to COVID-19? What impact has COVID-19 had on the delivery of preventive services in CHCs? Which PBRN services (e.g., data surveillance, training, evidence synthesis) are most impactful to real-world practices? What decision-making strategies were used in the PBRN and its practices to make real-time changes in response to the pandemic? What critical factors in successfully and sustainably transforming primary care are illuminated by pandemic-driven changes? DISCUSSION AND CONCLUSIONS: PBRNs enable real-world evaluation of practice change and natural experiments, and thus are ideal laboratories for implementation science research. We present a real-time example of how a PBRN Implementation Laboratory activated a response to study a historic natural experiment, to help other PBRNs charting a course through this pandemic.

Health Equity and the Tripartite Mission: Moving From Academic Health Centers to Academic-Community Health Systems.

Academic health centers (AHCs) play a significant role in educating the health care workforce, conducting innovative biomedical and clinical research, and delivering high-quality patient care. Much work remains, however, to adequately address the social determinants of health and equity that affect communities where patients live, work, and play. Doing so will help achieve the Quadruple Aim while addressing the unjust social structures that disproportionately impact communities of color and vulnerable populations. AHCs have a timely opportunity to focus their leading roles in education, research, and clinical care on social determinants, moving outside their walls to create academic-community health systems: a collection of academic-community partnerships advancing health equity through collaboration, power sharing, and cocreation.This Perspective proposes four strategies to start developing academic-community health systems. First, embark on all efforts through cocreation with communities. Second, address how future health care professionals are recruited. Third, build the right skills and opportunities for health care professionals to address health inequities. Finally, develop research agendas to evaluate programs addressing inequities. A fully realized vision of an academic-community health system will demonstrate interdependence between AHCs and the community. While considerable AHC resources are invested in building community capacity to improve health and health equity, health systems will also benefit in a multitude of ways, including increasing the diversity of ideas and experiences integrated into health systems. These strategies will support AHCs to embed across each arm of the tripartite mission a focus on partnering with communities to advance health equity together.

Progress towards using community context with clinical data in primary care.

Community-level factors have significant impacts on health. There is renewed enthusiasm for integrating these data with electronic health record (EHR) data for use in primary care to improve health equity in the USA. Thus, it is valuable to reflect on what has been published to date. Specifically, we comment on: (1) recommendations about combining community-level factors in EHRs for use in primary care; (2) examples of how these data have been combined and used; and (3) the impact of using combined data on healthcare, patient health and health equity. We found publications discussing the potential of combined data to inform clinical care, target interventions, track population health and spark community partnerships with the goal of reducing health disparities and improving health equity. Although there is great enthusiasm and potential for using these data to inform primary care, there is little evidence of improved healthcare, patient health or health equity.

Building Meaningful Patient Engagement in Research: Case Study From ADVANCE Clinical Data Research Network.

BACKGROUND: Strategies to engage patients to improve and enhance research and clinical care are increasingly being implemented in the United States, yet little is known about best practices for or the impacts of meaningful patient engagement. OBJECTIVE: We describe and reflect on our patient stakeholder groups, engagement framework, experiences, and lessons learned in engaging patients in research, from generating proposal ideas to disseminating findings. SETTING: The ADVANCE (Accelerating Data Value Across a National Community Health Center Network) clinical data research network is the nation's largest clinical dataset on the safety net, with outpatient clinical data from 122 health systems (1109 clinics) in 23 states. RESULTS: Patients stakeholders codeveloped the ADVANCE engagement framework and its implementation in partnership with network leaders. In phase I of ADVANCE, patients were involved with designing studies (input on primary outcome measures and methods) and usability testing (of the patient portal). In phase II, the network is prioritizing research training, dissemination opportunities, an "ambassador" program to pair more experienced patient stakeholders with those less experienced, and evaluation of engagement activities and impacts. DISCUSSION: The ADVANCE framework for patient engagement has successfully involved a diverse group of patients in the design, implementation, and interpretation of comparative effectiveness research. Our experience and framework can be used by other organizations and research networks to support patient engagement activities.

Reporting From the Front Lines: Implementing Oregon's Alternative Payment Methodology in Federally Qualified Health Centers.

Alternative payment models have been proposed as a way to facilitate patient-centered medical home model implementation, yet little is known about how payment reform translates into changes in care delivery. We conducted site visits, observed operations, and conducted interviews within 3 Federally Qualified Health Center organizations that were part of Oregon's Alternative Payment Methodology demonstration project. Data were analyzed using an immersion-crystallization approach. We identified several care delivery changes during the early stages of implementation, as well as challenges associated with this new model of payment. Future research is needed to further understand the implications of these changes.

Designing Health Information Technology Tools to Prevent Gaps in Public Health Insurance.

BACKGROUND: Changes in health insurance policies have increased coverage opportunities, but enrollees are required to annually reapply for benefits which, if not managed appropriately, can lead to insurance gaps. Electronic health records (EHRs) can automate processes for assisting patients with health insurance enrollment and re-enrollment. OBJECTIVE: We describe community health centers' (CHC) workflow, documentation, and tracking needs for assisting families with insurance application processes, and the health information technology (IT) tool components that were developed to meet those needs. METHOD: We conducted a qualitative study using semi-structured interviews and observation of clinic operations and insurance application assistance processes. Data were analyzed using a grounded theory approach. We diagramed workflows and shared information with a team of developers who built the EHR-based tools. RESULTS: Four steps to the insurance assistance workflow were common among CHCs: 1) Identifying patients for public health insurance application assistance; 2) Completing and submitting the public health insurance application when clinic staff met with patients to collect requisite information and helped them apply for benefits; 3) Tracking public health insurance approval to monitor for decisions; and 4) assisting with annual health insurance reapplication. We developed EHR-based tools to support clinical staff with each of these steps. CONCLUSION: CHCs are uniquely positioned to help patients and families with public health insurance applications. CHCs have invested in staff to assist patients with insurance applications and help prevent coverage gaps. To best assist patients and to foster efficiency, EHR based insurance tools need comprehensive, timely, and accurate health insurance information.

Protocol for the analysis of a natural experiment on the impact of the Affordable Care Act on diabetes care in community health centers.

BACKGROUND: It is hypothesized that Affordable Care Act (ACA) Medicaid expansions could substantially improve access to health insurance and healthcare services for patients at risk for diabetes mellitus (DM), with pre-DM, or already diagnosed with DM. The ACA called for every state to expand Medicaid coverage by 2014. In a 2012 legal challenge, the US Supreme Court ruled that states were not required to implement Medicaid expansions. This 'natural experiment' presents a unique opportunity to learn whether and to what extent Medicaid expansion can affect healthcare access and services for patients with DM risk, pre-DM, or DM. METHODS/DESIGN: Data from electronic health records (EHRs) from the Accelerating Data Value Across a National Community Health Center Network (ADVANCE) clinical data research network, which has data from >700 community health centers (CHCs), was included in the study. EHR data will be linked to Oregon Medicaid claims data. Data collection will include information on changes in health insurance, service receipt, and health outcomes, spanning 9 years (pre- and post-expansion), comparing states that expanded Medicaid, and those that did not. Patients included in this study will be diagnosed with DM, be at risk for DM, or have pre-DM, between the ages of 19 and 64, with ≥1 ambulatory visit. Sample size is estimated to be roughly 275,000 patients. Biostatistical analyses will include the difference-in-differences (DID) methodology and a generalized linear mixed model. Econometric analyses will include a DID two-part method to calculate the difference in Medicaid expenditures in Oregon among newly insured CHC patients. DISCUSSION: Findings will have national relevance on DM health services and outcomes and will be shared through national conferences and publications. The findings will provide information needed to impact the policy as it is related to access to health insurance and receipt of healthcare among a vulnerable population. TRIAL REGISTRATION: This project is registered with ClinicalTrials.gov ( NCT02685384 ). Registered 18 May 2016.

Creating 21st-Century Laboratories and Classrooms for Improving Population Health: A Call to Action for Academic Medical Centers.

Academic medical centers (AMCs) in the United States built world-class infrastructure to successfully combat disease in the 20th century, which is inadequate for the complexity of sustaining and improving population health. AMCs must now build first-rate 21st-century infrastructure to connect combating disease and promoting health. This infrastructure must acknowledge the bio-psycho-social-environmental factors impacting health and will need to reach far beyond the AMC walls to foster community "laboratories" that support the "science of health," complementary to those supporting the "science of medicine"; cultivate community "classrooms" to stimulate learning and discovery in the places where people live, work, and play; and strengthen bridges between academic centers and these community laboratories and classrooms to facilitate bidirectional teaching, learning, innovation, and discovery.Private and public entities made deep financial investments that contributed to the AMC disease-centered approach to clinical care, education, and research in the 20th century. Many of these same funders now recognize the need to transform U.S. health care into a system that is accountable for population health and the need for a medical workforce equipped with the skills to measure and improve health. Innovative ideas about communities as centers of learning, the importance of social factors as major determinants of health, and the need for multidisciplinary perspectives to solve complex problems are not new; many are 20th-century ideas still waiting to be fully implemented. The window of opportunity is now. The authors articulate how AMCs must take bigger and bolder steps to become leaders in population health.

Testing health information technology tools to facilitate health insurance support: a protocol for an effectiveness-implementation hybrid randomized trial.

BACKGROUND: Patients with gaps in health insurance coverage often defer or forgo cancer prevention services. These delays in cancer detection and diagnoses lead to higher rates of morbidity and mortality and increased costs. Recent advances in health information technology (HIT) create new opportunities to enhance insurance support services that reduce coverage gaps through automated processes applied in healthcare settings. This study will assess the implementation of insurance support HIT tools and their effectiveness at improving patients' insurance coverage continuity and cancer screening rates. METHODS/DESIGN: This study uses a hybrid cluster-randomized design-a combined effectiveness and implementation trial-in community health centers (CHCs) in the USA. Eligible CHC clinic sites will be randomly assigned to one of two groups in the trial's implementation component: tools + basic training (Arm I) and tools + enhanced training + facilitation (Arm II). A propensity score-matched control group of clinics will be selected to assess the tools' effectiveness. Quantitative analyses of the tools' impact will use electronic health record and Medicaid data to assess effectiveness. Qualitative data will be collected to evaluate the implementation process, understand how the HIT tools are being used, and identify facilitators and barriers to their implementation and use. DISCUSSION: This study will test the effectiveness of HIT tools to enhance insurance support in CHCs and will compare strategies for facilitating their implementation in "real-world" practice settings. Findings will inform further development and, if indicated, more widespread implementation of insurance support HIT tools. TRIAL REGISTRATION: Clinical trial NTC02355262.

Improve Synergy Between Health Information Exchange and Electronic Health Records to Increase Rates of Continuously Insured Patients.

INTRODUCTION: The Affordable Care Act increases health insurance options, yet many Americans may struggle to consistently maintain coverage. While health care providers have traditionally not been involved in providing insurance enrollment support to their patients, the ability for them to do so now exists. We propose that providers could capitalize on the expansion of electronic health records (EHRs) and the advances in health information exchanges (HIEs) to improve their patients' insurance coverage rates and continuity. EVIDENCE FOR ARGUMENT: We describe a project in which we are building strategies for linking, and thus improving synergy between, payer and EHR data. Through this effort, care teams will have access to new automated tools and increased EHR functionality designed to help them assist their patients in obtaining and maintaining health insurance coverage. SUGGESTION FOR THE FUTURE: The convergence of increasing EHR adoption, improving HIE functionality, and expanding insurance coverage options, creates new opportunities for clinics to help their patients obtain public health insurance. Harnessing this nascent ability to exchange information between payers and providers may improve synergies between HIE and EHRs, and thus support clinic-based efforts to keep patients continuously insured.

Lessons Learned from Developing a Patient Engagement Panel: An OCHIN Report.

There is renewed interest in patient engagement in clinical and research settings, creating a need for documenting and publishing lessons learned from efforts to meaningfully engage patients. This article describes early lessons learned from the development of OCHIN's Patient Engagement Panel (PEP). OCHIN supports a national network of more than 300 community health centers (CHCs) and other primary care settings that serve over 1.5 million patients annually across nearly 20 states. The PEP was conceived in 2009 to harness the CHC tradition of patient engagement in this new era of patient-centered outcomes research and to ensure that patients were engaged throughout the life cycle of our research projects, from conception to dissemination. Developed by clinicians and researchers within our practice-based research network, recruitment of patients to serve as PEP members began in early 2012. The PEP currently has a membership of 18 patients from 3 states. Over the past 24 months, the PEP has been involved with 12 projects. We describe developing the PEP and challenges and lessons learned (eg, recruitment, funding model, creating value for patient partners, compensation). These lessons learned are relevant not only for research but also for patient engagement in quality improvement efforts and other clinical initiatives.

IMPACCT Kids' Care: a real-world example of stakeholder involvement in comparative effectiveness research.

The Patient-Centered Outcomes Research Institute has accelerated conversations about the importance of actively engaging stakeholders in all aspects of comparative effectiveness research (CER). Other scientific disciplines have a history of stakeholder engagement, yet few empirical examples exist of how these stakeholders can inform and enrich CER. Here we present a case study which includes the methods used to engage stakeholders, what we learned from them, and how we incorporated their ideas in a CER project. We selected stakeholders from key groups, built relationships with them and collected their feedback through interviews, observation and ongoing meetings during the four research process phases: proposal development, adapting study methods, understanding the context and information technology tool design and refinement.

Innovative methods for parents and clinics to create tools for kids' care (IMPACCT Kids' Care) study protocol.

BACKGROUND: Despite expansions in public health insurance, many children remain uninsured or experience gaps in coverage. Community health centers (CHCs) provide primary care to many children at risk for uninsurance and are well-positioned to help families obtain and retain children's coverage. Recent advances in health information technology (HIT) capabilities provide the means to create tools that could enhance CHCs' insurance outreach efforts. OBJECTIVE: To present the study design, baseline patient characteristics, variables, and statistical methods for the Innovative Methods for Parents And Clinics to Create Tools for Kids' Care (IMPACCT Kids' Care) study. METHODS/DESIGN: In this mixed methods study, we will design, test and refine health insurance outreach HIT tools through a user-centered process. We will then implement the tools in four CHCs and evaluate their effectiveness and barriers and facilitators to their implementation. To measure effectiveness, we will quantitatively assess health insurance coverage continuity and utilization of healthcare services for pediatric patients in intervention CHCs compared to matched control sites using electronic health record (EHR) and Oregon Medicaid administrative data over 18months pre- and 18months post-implementation (n=34,867 children). We will also qualitatively assess the implementation process to understand how the tools fit into the clinics' workflows and the CHC staff experiences with the tools. CONCLUSIONS: This study creates, implements, and evaluates health insurance outreach HIT tools. The use of such tools will likely improve care delivery and health outcomes, reduce healthcare disparities for vulnerable populations, and enhance overall healthcare system performance. ClinicalTrials.gov Identifier: NCT02298361.