Getting cozy with causality: Advances to the causal pathway diagramming method to enhance implementation precision.

Background: Implementation strategies are theorized to work well when carefully matched to implementation determinants and when factors-preconditions, moderators, etc.-that influence strategy effectiveness are prospectively identified and addressed. Existing methods for strategy selection are either imprecise or require significant technical expertise and resources, undermining their utility. This article outlines refinements to causal pathway diagrams (CPDs), a method for articulating the causal process through which implementation strategies work and offers illustrations of their use. Method: CPDs are a visualization tool to represent an implementation strategy, its mechanism(s) (i.e., the processes through which a strategy is thought to operate), determinants it is intended to address, factors that may impede or facilitate its effectiveness, and the series of outcomes that should be expected if the strategy is operating as intended. We offer principles for constructing CPDs and describe their key functions. Results: Applications of the CPD method by study teams from two National Institute of Health-funded Implementation Science Centers and a research grant are presented. These include the use of CPDs to (a) match implementation strategies to determinants, (b) understand the conditions under which an implementation strategy works, and (c) develop causal theories of implementation strategies. Conclusions: CPDs offer a novel method for implementers to select, understand, and improve the effectiveness of implementation strategies. They make explicit theoretical assumptions about strategy operation while supporting practical planning. Early applications have led to method refinements and guidance for the field.

Advances to the Causal Pathway Diagramming Method to Enhance Implementation Precision Plain Language Summary Implementation strategies often fail to produce meaningful improvements in the outcomes we hope to impact. Better tools for choosing, designing, and evaluating implementation strategies may improve their performance. We developed a tool, causal pathway diagrams (CPD), to visualize and describe how implementation strategies are expected to work. In this article, we describe refinements to the CPD tool and accompanying approach. We use real illustrations to show how CPDs can be used to improve how to match strategies to barriers, understand the conditions in which those strategies work best, and develop generalizable theories describing how implementation strategies work. CPDs can serve as both a practical and scientific tool to improve the planning, deployment, and evaluation of implementation strategies. We demonstrate the range of ways that CPDs are being used, from a highly practical tool to improve implementation practice to a scientific approach to advance testing and theorizing about implementation strategies.

Sitting Time Reduction and Blood Pressure in Older Adults: A Randomized Clinical Trial.

Importance: Practical health promotion strategies for improving cardiometabolic health in older adults are needed. Objective: To examine the efficacy of a sedentary behavior reduction intervention for reducing sitting time and improving blood pressure in older adults. Design, Setting, and Participants: This parallel-group randomized clinical trial was conducted in adults aged 60 to 89 years with high sitting time and body mass index of 30 to 50 from January 1, 2019, to November 31, 2022, at a health care system in Washington State. Intervention: Participants were randomized 1:1 to the sitting reduction intervention or a healthy living attention control condition for 6 months. Intervention participants received 10 health coaching contacts, sitting reduction goals, and a standing desk and fitness tracker to prompt sitting breaks. The attention control group received 10 health coaching contacts to set general healthy living goals, excluding physical activity or sedentary behavior. Main Outcomes and Measures: The primary outcome, measured at baseline, 3 months, and 6 months, was sitting time assessed using accelerometers worn for 7 days at each time point. Coprimary outcomes were systolic and diastolic blood pressure measured at baseline and 6 months. Results: A total of 283 participants (140 intervention and 143 control) were randomized (baseline mean [SD] age, 68.8 [6.2] years; 186 [65.7%] female; mean [SD] body mass index, 34.9 [4.7]). At baseline, 147 (51.9%) had a hypertension diagnosis and 97 (69.3%) took at least 1 antihypertensive medication. Sitting time was reduced, favoring the intervention arm, with a difference in the mean change of -31.44 min/d at 3 months (95% CI, -48.69 to -14.19 min/d; P < .001) and -31.85 min/d at 6 months (95% CI, -52.91 to -10.79 min/d; P = .003). Systolic blood pressure change was lower by 3.48 mm Hg, favoring the intervention arm at 6 months (95% CI, -6.68 to -0.28 mm Hg; P = .03). There were 6 serious adverse events in each arm and none were study related. Conclusions and Relevance: In this study of a 6-month sitting reduction intervention, older adults in the intervention reduced sedentary time by more than 30 min/d and reduced systolic blood pressure. Sitting reduction could be a promising approach to improve health in older adults. Trial Registration: ClinicalTrials.gov Identifier: NCT03739762.

Study protocol: Novel Methods for Implementing Measurement-Based Care with youth in Low-Resource Environments (NIMBLE).

BACKGROUND: For youth receiving care in community mental health centers, comorbidities are the rule rather than the exception. Using measurement-based care (MBC), or the routine evaluation of symptoms to inform care decisions, as the foundation of treatment for youth with comorbid problems significantly improves the impact of psychotherapy by focusing care and building engagement and alliance. MBC increases the rate of symptom improvement, detects clients who would otherwise deteriorate, and alerts clinicians to non-responders. Despite its demonstrated utility, MBC is rarely implemented with fidelity; less than 15% of providers report using MBC per recommendations. Previous efforts to support MBC implementation have yielded suboptimal outcomes, in part, due to organizations' challenges with identifying and prioritizing barriers and selecting and developing strategies to overcome them. New methods are needed for identifying and prioritizing barriers, and matching strategies to barriers to optimize MBC implementation and treatment quality to improve youth mental health outcomes in community settings. METHODS: Pragmatic implementation methods will be piloted in four diverse community mental health centers. Methods include (a) rapid evidence synthesis; (b) rapid ethnography; (c) design kits (e.g., kits with disposable cameras, journals, maps); (d) barrier prioritization, and (e) causal pathway diagramming. These activities will generate actionable barriers; subsequently, we will use facilitated group processes to prioritize barriers and develop causal pathway diagrams to match strategies to barriers to create implementation plans that optimize MBC fidelity (Aim 1). We will track strategy deployment for 6 months, then compare MBC fidelity for another 6 months post-implementation with data from 2 years of historical controls (Aim 2). Finally, we will co-design a toolkit for design kit methods with youth and the practice and scientific communities (Aim 3). DISCUSSION: Optimizing MBC implementation in community mental health centers could transform youth mental health care by ensuring the most pressing symptoms are targeted early in treatment. The discussion section highlights expected challenges and limits to using the five methods, including recruitment and engagement given the high pressure on community mental health settings. TRIAL REGISTRATION: Clinicaltrials.gov. NCT05644756 . Registered on 18 November 2022. This trial was retrospectively registered.

Exploring Differences in Older Adult Accelerometer-Measured Sedentary Behavior and Resting Blood Pressure Before and During the COVID-19 Pandemic.

Older adults have higher sedentary behavior (SB), lower physical activity, and are particularly susceptible to negative impacts from the COVID-19 pandemic and associated public health restrictions. Pandemic impacts to SB and health, particularly via objective assessment, are not well documented in the literature. Here we described differences in SB, physical activity, and blood pressure (BP) for older adults before and during the pandemic. Baseline thigh-worn activPAL accelerometer and BP measurements from 95 participants enrolled in a SB intervention trial pre-pandemic were compared to 60 enrolled post-pandemic. We used linear regression models adjusted for demographic and health factors to estimate differences in sample means of SB measures and BP. The post-COVID sample was older (age 67 vs. 70), more female (60% vs. 72%), and included more individuals of color (21% vs. 32%). In fully adjusted models, systolic BP was statistically significantly higher in the post-COVID group (6.8, 95% CI: [0.3,13.3]). After adjustment, activPAL-measured and self-reported activity were non-significant but trended towards greater total sitting (0.4 hours [-0.3, 1.1]), fewer daily steps (-270 [-1078, 538]), and greater self-reported TV time (0.4 hours, [-0.3, 1.1]) post-COVID. Future analyses are warranted to better quantify these impacts and guide clinical care and future interventions.

A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence.

Background: Measurement is a critical component for any field. Systematic reviews are a way to locate measures and uncover gaps in current measurement practices. The present study identified measures used in behavioral health settings that assessed all constructs within the Process domain and two constructs from the Inner setting domain as defined by the Consolidated Framework for Implementation Research (CFIR). While previous conceptual work has established the importance social networks and key stakeholders play throughout the implementation process, measurement studies have not focused on investigating the quality of how these activities are being carried out. Methods: The review occurred in three phases: Phase I, data collection included (1) search string generation, (2) title and abstract screening, (3) full text review, (4) mapping to CFIR-constructs, and (5) "cited-by" searches. Phase II, data extraction, consisted of coding information relevant to the nine psychometric properties included in the Psychometric And Pragmatic Rating Scale (PAPERS). In Phase III, data analysis was completed. Results: Measures were identified in only seven constructs: Structural characteristics (n = 13), Networks and communication (n = 29), Engaging (n = 1), Opinion leaders (n = 5), Champions (n = 5), Planning (n = 5), and Reflecting and evaluating (n = 5). No quantitative assessment measures of Formally appointed implementation leaders, External change agents, or Executing were identified. Internal consistency and norms were reported on most often, whereas no studies reported on discriminant validity or responsiveness. Not one measure in the sample reported all nine psychometric properties evaluated by the PAPERS. Scores in the identified sample of measures ranged from "-2" to "10" out of a total of "36." Conclusions: Overall measures demonstrated minimal to adequate evidence and available psychometric information was limited. The majority were study specific, limiting their generalizability. Future work should focus on more rigorous measure development and testing of currently existing measures, while moving away from creating new, single use measures. Plain Language Summary: How we measure the processes and players involved for implementing evidence-based interventions is crucial to understanding what factors are helping or hurting the intervention's use in practice and how to take the intervention to scale. Unfortunately, measures of these factors-stakeholders, their networks and communication, and their implementation activities-have received little attention. This study sought to identify and evaluate the quality of these types of measures. Our review focused on collecting measures used for identifying influential staff members, known as opinion leaders and champions, and investigating how they plan, execute, engage, and evaluate the hard work of implementation. Upon identifying these measures, we collected all published information about their uses to evaluate the quality of their evidence with respect to their ability to produce consistent results across items within each use (i.e., reliable) and if they assess what they are intending to measure (i.e., valid). Our searches located over 40 measures deployed in behavioral health settings for evaluation. We observed a dearth of evidence for reliability and validity and when evidence existed the quality was low. These findings tell us that more measurement work is needed to better understand how to optimize players and processes for the purposes of successful implementation.

The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation.

To rigorously measure the implementation of evidence-based interventions, implementation science requires measures that have evidence of reliability and validity across different contexts and populations. Measures that can detect change over time and impact on outcomes of interest are most useful to implementers. Moreover, measures that fit the practical needs of implementers could be used to guide implementation outside of the research context. To address this need, our team developed a rating scale for implementation science measures that considers their psychometric and pragmatic properties and the evidence available. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) can be used in systematic reviews of measures, in measure development, and to select measures. PAPERS may move the field toward measures that inform robust research evaluations and practical implementation efforts.

Transitioning patients from outpatient mental health services to primary care: A rapid literature review.

Background: A lack of access to mental health services is a critical barrier to obtaining evidence-based care. One strategy to improve access is to transition stable patients out of mental health specialty services and into primary care, thus opening availability for new patients and those with acute mental health needs. To support these transitions, organizations might explore a range of new practices and implementation strategies. Methods: We conducted a rapid literature review to summarize descriptions from the research literature about practices for transitioning stable patients from outpatient mental health services to primary care, as well as implementation strategies to enhance the adoption and sustainment of these practices. We searched PsycINFO and Cumulated Index to Nursing and Allied Health Literature (CINAHL) for articles published between January 2000 and August 2019. For articles meeting inclusion criteria, we abstracted data on study characteristics, transition practices, and implementation strategies. Results: We included 11 articles representing diverse study designs, settings, and health care organizations. Across these articles, we identified six categories of commonly described transition practices, with patient engagement appearing the most frequently (10 articles), followed by shared treatment planning (eight articles), assessment of recovery and stability, care coordination, follow up and support, and medication management (seven articles each). Less frequently, articles included descriptions of implementation strategies, with five articles describing efforts to train and educate stakeholders and four articles describing the use of evaluative and iterative strategies. Conclusions: We identified descriptions of several common practices to help patients transition from mental health specialty services to primary care, but there are opportunities for an increased focus on implementation strategies to enhance the adoption and sustainment of these transition practices. More research is needed to better understand the effectiveness of specific transition interventions and the feasibility of deploying these interventions in heterogeneous health care settings.

"They're Going to Zoom It": A Qualitative Investigation of Impacts and Coping Strategies During the COVID-19 Pandemic Among Older Adults.

Introduction: Older adults, who already have higher levels of social isolation, loneliness, and sedentary behavior, are particularly susceptible to negative impacts from social distancing mandates meant to control the spread of COVID-19. We sought to explore the physical, mental, and social health impacts of the pandemic on older adults and their coping techniques. Materials and Methods: We conducted 25 semi-structured interviews with a sub-sample of participants in an ongoing sedentary behavior reduction intervention. Interviews were recorded and transcribed, and iterative coding was used to extract key themes. Results: Most participants reported an increase in sedentary behavior due to limitations on leaving their home and increased free time to pursue seated hobbies (e.g., reading, knitting, tv). However, many participants also reported increased levels of intentional physical activity and exercise, particularly outdoors or online. Participants also reported high levels of stress and a large decrease in in-person social connection. Virtual connection with others through phone and video was commonly used to stay connected with friends and family, engage in community groups and activities, and cope with stress and social isolation. Maintenance of a positive attitude and perspective gained from past hardships was also an important coping strategy for many participants. Discussion: The COVID-19 pandemic and associated social distancing measures have impacted older adults' perceived levels of activity, stress, and social isolation, but many leveraged technology and prior life experiences to cope. These themes could inform future interventions for older adults dealing with chronic stress and isolation.

Self-reported Access to Firearms Among Patients Receiving Care for Mental Health and Substance Use.

Importance: Firearms are the most common method of suicide, one of the "diseases of despair" driving increased mortality in the US over the past decade. However, routine standardized questions about firearm access are uncommon, particularly among adult populations, who are more often asked at the discretion of health care clinicians. Because standard questions are rare, patterns of patient-reported access are unknown. Objective: To evaluate whether and how patients self-report firearm access information on a routine mental health monitoring questionnaire and additionally to examine sociodemographic and clinical associations of reported access. Design Setting and Participants: Cross-sectional study of patients receiving care for mental health and/or substance use in primary care or outpatient mental health specialty clinics of Kaiser Permanente Washington, an integrated health insurance provider and care delivery system. Main Outcomes and Measures: Electronic health records were used to identify patients who completed a standardized self-reported mental health monitoring questionnaire after a single question about firearm access was added from January 1, 2016, through December 31, 2019. Primary analyses evaluated response (answered vs not answered) and reported access (yes vs no) among those who answered, separately for patients seen in primary care and mental health. These analyses also evaluated associations between patient characteristics and reported firearm access. Data analysis took place from February 2020 through May 2021. Results: Among patients (n = 128 802) who completed a mental health monitoring questionnaire during the study period, 74.4% (n = 95 875) saw a primary care clinician and 39.3% (n = 50 631) saw a mental health specialty clinician. The primary care and mental health samples were predominantly female (63.1% and 64.9%, respectively) and White (75.7% and 77.0%), with a mean age of 42.8 and 51.1 years. In primary care, 83.4% of patients answered the question about firearm access, and 20.9% of patients who responded to the firearm question reported having access. In mental health, 91.8% of patients answered the question, and 15.3% reported having access. Conclusions and Relevance: In this cross-sectional study of adult patients receiving care for mental health and substance use, most patients answered a question about firearm access on a standardized mental health questionnaire. These findings provide a critical foundation to help advance understanding of the utility of standardized firearm access assessment and to inform development of practice guidelines and recommendations. Responses to standard firearm access questions used in combination with dialogue and decision-making resources about firearm access and storage may improve suicide prevention practices and outcomes.

Protocol for a randomized controlled trial of sitting reduction to improve cardiometabolic health in older adults.

Older adults with obesity spend the majority of their waking hours sedentary. Given substantial barriers to regular physical activity in this population, approaches to reduce sedentary time could be an effective health promotion strategy. We present the protocol of a randomized controlled trial to reduce sitting time in older adults with a body mass index of 30 kg/m2 or above. Participants (N = 284) will be randomized to receive a sitting reduction intervention (termed I-STAND) or a healthy living focused attention control condition. I-STAND includes 10 contacts with a health coach (10 sessions total) and participants receive a wrist-worn prompting device and portable standing desk. The healthy living condition includes 10 sessions with a health coach to set goals around various topics relating to healthy aging. Participants receive their assigned intervention for 6 months. After 6 months, those receiving the I-STAND condition are re-randomized to receive five booster health coaching sessions by 'phone or no further contact; healthy living participants receive no further contact and those in both conditions are followed for an additional 6 months. Measurements initially included wearing an activPAL device and completing several biometric tests (e.g., blood pressure, HbA1c), at baseline, 3 months, 6 months, and 12 months; however, during the COVID-19 pandemic we shifted to remote assessments and were unable to collect all of these measures. The primary outcomes remained activPAL-assessed sitting time and blood pressure. Recruitment is anticipated to be completed in 2022.

Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS).

The use of reliable, valid measures in implementation practice will remain limited without pragmatic measures. Previous research identified the need for pragmatic measures, though the characteristic identification used only expert opinion and literature review. Our team completed four studies to develop a stakeholder-driven pragmatic rating criteria for implementation measures. We published Studies 1 (identifying dimensions of the pragmatic construct) and 2 (clarifying the internal structure) that engaged stakeholders-participants in mental health provider and implementation settings-to identify 17 terms/phrases across four categories: Useful, Compatible, Acceptable, and Easy. This paper presents Studies 3 and 4: a Delphi to ascertain stakeholder-prioritized dimensions within a mental health context, and a pilot study applying the rating criteria. Stakeholders (N = 26) participated in a Delphi and rated the relevance of 17 terms/phrases to the pragmatic construct. The investigator team further defined and shortened the list, which were piloted with 60 implementation measures. The Delphi confirmed the importance of all pragmatic criteria, but provided little guidance on relative importance. The investigators removed or combined terms/phrases to obtain 11 criteria. The 6-point rating system assigned to each criterion demonstrated sufficient variability across items. The grey literature did not add critical information. This work produced the first stakeholder-driven rating criteria to assess whether measures are pragmatic. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) combines the pragmatic criteria with psychometric rating criteria, from previous work. Use of PAPERS can inform development of implementation measures and to assess the quality of existing measures.

Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review.

Background: Organizational culture, organizational climate, and implementation climate are key organizational constructs that influence the implementation of evidence-based practices. However, there has been little systematic investigation of the availability of psychometrically strong measures that can be used to assess these constructs in behavioral health. This systematic review identified and assessed the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs as defined by the Consolidated Framework for Implementation Research (CFIR) and Ehrhart and colleagues. Methods: Data collection involved search string generation, title and abstract screening, full-text review, construct assignment, and citation searches for all known empirical uses. Data relevant to nine psychometric criteria from the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) were extracted: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. Extracted data for each criterion were rated on a scale from -1 ("poor") to 4 ("excellent"), and each measure was assigned a total score (highest possible score = 36) that formed the basis for head-to-head comparisons of measures for each focal construct. Results: We identified full measures or relevant subscales of broader measures for organizational culture (n = 21), organizational climate (n = 36), implementation climate (n = 2), tension for change (n = 2), compatibility (n = 6), relative priority (n = 2), organizational incentives and rewards (n = 3), goals and feedback (n = 3), and learning climate (n = 2). Psychometric evidence was most frequently available for internal consistency and norms. Information about other psychometric properties was less available. Median ratings for psychometric properties across categories of measures ranged from "poor" to "good." There was limited evidence of responsiveness or predictive validity. Conclusion: While several promising measures were identified, the overall state of measurement related to these constructs is poor. To enhance understanding of how these constructs influence implementation research and practice, measures that are sensitive to change and predictive of key implementation and clinical outcomes are required. There is a need for further testing of the most promising measures, and ample opportunity to develop additional psychometrically strong measures of these important constructs. Plain Language Summary: Organizational culture, organizational climate, and implementation climate can play a critical role in facilitating or impeding the successful implementation and sustainment of evidence-based practices. Advancing our understanding of how these contextual factors independently or collectively influence implementation and clinical outcomes requires measures that are reliable and valid. Previous systematic reviews identified measures of organizational factors that influence implementation, but none focused explicitly on behavioral health; focused solely on organizational culture, organizational climate, and implementation climate; or assessed the evidence base of all known uses of a measure within a given area, such as behavioral health-focused implementation efforts. The purpose of this study was to identify and assess the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs that have been used in behavioral health-focused implementation research. We identified 21 measures of organizational culture, 36 measures of organizational climate, 2 measures of implementation climate, 2 measures of tension for change, 6 measures of compatibility, 2 measures of relative priority, 3 measures of organizational incentives and rewards, 3 measures of goals and feedback, and 2 measures of learning climate. Some promising measures were identified; however, the overall state of measurement across these constructs is poor. This review highlights specific areas for improvement and suggests the need to rigorously evaluate existing measures and develop new measures.

A pilot study comparing tools for tracking implementation strategies and treatment adaptations.

Background: Tailoring implementation strategies and adapting treatments to better fit the local context may improve their effectiveness. However, there is a dearth of valid, reliable, pragmatic measures that allow for the prospective tracking of strategies and adaptations according to reporting recommendations. This study describes the development and pilot testing of three tools to be designed to serve this purpose. Methods: Measure development was informed by two systematic reviews of the literature (implementation strategies and treatment adaptation). The three resulting tools vary with respect to the degree of structure (brainstorming log = low, activity log = moderate, detailed tracking log = high). To prospectively track treatment adaptations and implementation strategies, three stakeholder groups (treatment developer, implementation practitioners, and mental health providers) were randomly assigned one tool per week through an anonymous web-based survey for 12 weeks and incentivized to participate. Three established implementation outcome measures, the Acceptability of Intervention Measure, Intervention Appropriateness Measure, and Feasibility of Intervention Measure, were used to assess the tools. Semi-structured interviews were conducted to gather more nuanced information from stakeholders regarding their perceptions of the tools and the tracking process. Results: The three tracking tools demonstrated moderate to good acceptability, appropriateness, and feasibility; the activity log was deemed the most feasible of the three tools. Implementation practitioners rated the tools the highest of the three stakeholder groups. The tools took an average of 15 min or less to complete. Conclusion: This study sought to fill methodological gaps that prevent stakeholders and researchers from discerning which strategies are most important to deploy for promoting implementation and sustainment of evidence-based practices. These tools would allow researchers and practitioners to track whether activities were treatment adaptations or implementation strategies and what barrier(s) each targets. These tools could inform prospective tailoring of implementation strategies and treatment adaptations, which would promote scale out and spread. Plain Language Summary: Strategies to support the implementation of evidence-based practices may be more successful if they are carefully customized based on local factors. Evidence-based practices themselves may be thoughtfully changed to better meet the needs of the settings and recipients. This study reports on a pilot study that aimed to create various types of tools to help individuals involved in implementation efforts track the actions they take to modify and implement interventions. These tools allow individuals to track the types of activities they are involved in, when the activities occurred, who was involved in the implementation efforts, and the reasons or rationale for the actions. The three tools in this study used a combination of open-ended and forced-response questions to test how the type of data recorded changed. Participants generally found the tools quick and easy to use and helpful in planning the delivery of an evidence-based practice. Most participants wanted more training in implementation science terminology and how to complete the tracking tools. Participating mental health providers would have liked more opportunities to review the data collected from the tools with their supervisors to use the data to improve the delivery of the evidence-based practice. These tools can help researchers, providers, and staff involved in implementation efforts to better understand what actions are needed to improve implementation success. Future research should address gaps identified in this study, such as the need to involve more participants in the tool development process.

Measuring readiness for implementation: A systematic review of measures' psychometric and pragmatic properties.

Background: Systematic measure reviews can facilitate advances in implementation research and practice by locating reliable, valid, pragmatic measures; identifying promising measures needing refinement and testing; and highlighting measurement gaps. This review identifies and evaluates the psychometric and pragmatic properties of measures of readiness for implementation and its sub-constructs as delineated in the Consolidated Framework for Implementation Research: leadership engagement, available resources, and access to knowledge and information. Methods: The systematic review methodology is described fully elsewhere. The review, which focused on measures used in mental or behavioral health, proceeded in three phases. Phase I, data collection, involved search string generation, title and abstract screening, full text review, construct assignment, and cited citation searches. Phase II, data extraction, involved coding relevant psychometric and pragmatic information. Phase III, data analysis, involved two trained specialists independently rating each measure using Psychometric and Pragmatic Evidence Rating Scales (PAPERS). Frequencies and central tendencies summarized information availability and PAPERS ratings. Results: Searches identified 9 measures of readiness for implementation, 24 measures of leadership engagement, 17 measures of available resources, and 6 measures of access to knowledge and information. Information about internal consistency was available for most measures. Information about other psychometric properties was often not available. Ratings for internal consistency were "adequate" or "good." Ratings for other psychometric properties were less than "adequate." Information on pragmatic properties was most often available regarding cost, language readability, and brevity. Information was less often available regarding training burden and interpretation burden. Cost and language readability generally exhibited "good" or "excellent" ratings, interpretation burden generally exhibiting "minimal" ratings, and training burden and brevity exhibiting mixed ratings across measures. Conclusion: Measures of readiness for implementation and its sub-constructs used in mental health and behavioral health care are unevenly distributed, exhibit unknown or low psychometric quality, and demonstrate mixed pragmatic properties. This review identified a few promising measures, but targeted efforts are needed to systematically develop and test measures that are useful for both research and practice. Plain language abstract: Successful implementation of effective mental health or behavioral health treatments in service delivery settings depends in part on the readiness of the service providers and administrators to implement the treatment; the engagement of organizational leaders in the implementation effort; the resources available to support implementation, such as time, money, space, and training; and the accessibility of knowledge and information among service providers about the treatment and how it works. It is important that the methods for measuring these factors are dependable, accurate, and practical; otherwise, we cannot assess their presence or strength with confidence or know whether efforts to increase their presence or strength have worked. This systematic review of published studies sought to identify and evaluate the quality of questionnaires (referred to as measures) that assess readiness for implementation, leadership engagement, available resources, and access to knowledge and information. We identified 56 measures of these factors and rated their quality in terms of how dependable, accurate, and practical they are. Our findings indicate there is much work to be done to improve the quality of available measures; we offer several recommendations for doing so.

Quantitative measures of health policy implementation determinants and outcomes: a systematic review.

BACKGROUND: Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized. Implementation outcomes include acceptability, adoption, appropriateness, compliance/fidelity, feasibility, penetration, sustainability, and costs. This systematic review identified quantitative measures that are used to assess health policy implementation determinants and outcomes and evaluated the quality of these measures. METHODS: Three frameworks guided the review: Implementation Outcomes Framework (Proctor et al.), Consolidated Framework for Implementation Research (Damschroder et al.), and Policy Implementation Determinants Framework (Bullock et al.). Six databases were searched: Medline, CINAHL Plus, PsycInfo, PAIS, ERIC, and Worldwide Political. Searches were limited to English language, peer-reviewed journal articles published January 1995 to April 2019. Search terms addressed four levels: health, public policy, implementation, and measurement. Empirical studies of public policies addressing physical or behavioral health with quantitative self-report or archival measures of policy implementation with at least two items assessing implementation outcomes or determinants were included. Consensus scoring of the Psychometric and Pragmatic Evidence Rating Scale assessed the quality of measures. RESULTS: Database searches yielded 8417 non-duplicate studies, with 870 (10.3%) undergoing full-text screening, yielding 66 studies. From the included studies, 70 unique measures were identified to quantitatively assess implementation outcomes and/or determinants. Acceptability, feasibility, appropriateness, and compliance were the most commonly measured implementation outcomes. Common determinants in the identified measures were organizational culture, implementation climate, and readiness for implementation, each aspects of the internal setting. Pragmatic quality ranged from adequate to good, with most measures freely available, brief, and at high school reading level. Few psychometric properties were reported. CONCLUSIONS: Well-tested quantitative measures of implementation internal settings were under-utilized in policy studies. Further development and testing of external context measures are warranted. This review is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and determinants to help practitioners and researchers spread evidence-informed policies to improve population health. REGISTRATION: Not registered.

Psychometric and Pragmatic Properties of Social Risk Screening Tools: A Systematic Review.

CONTEXT: Health systems increasingly are exploring implementation of standardized social risk assessments. Implementation requires screening tools both with evidence of validity and reliability (psychometric properties) and that are low cost, easy to administer, readable, and brief (pragmatic properties). These properties for social risk assessment tools are not well understood and could help guide selection of assessment tools and future research. EVIDENCE ACQUISITION: The systematic review was conducted during 2018 and included literature from PubMed and CINAHL published between 2000 and May 18, 2018. Included studies were based in the U.S., included tools that addressed at least 2 social risk factors (economic stability, education, social and community context, healthcare access, neighborhood and physical environment, or food), and were administered in a clinical setting. Manual literature searching was used to identify empirical uses of included screening tools. Data on psychometric and pragmatic properties of each tool were abstracted. EVIDENCE SYNTHESIS: Review of 6,838 unique citations yielded 21 unique screening tools and 60 articles demonstrating empirical uses of the included screening tools. Data on psychometric properties were sparse, and few tools reported use of gold standard measurement development methods. Review of pragmatic properties indicated that tools were generally low cost, written for low-literacy populations, and easy to administer. CONCLUSIONS: Multiple low-cost, low literacy tools are available for social risk screening in clinical settings, but psychometric data are very limited. More research is needed on clinic-based screening tool reliability and validity as these factors should influence both adoption and utility. SUPPLEMENT INFORMATION: This article is part of a supplement entitled Identifying and Intervening on Social Needs in Clinical Settings: Evidence and Evidence Gaps, which is sponsored by the Agency for Healthcare Research and Quality of the U.S. Department of Health and Human Services, Kaiser Permanente, and the Robert Wood Johnson Foundation.

An updated protocol for a systematic review of implementation-related measures.

BACKGROUND: Implementation science is the study of strategies used to integrate evidence-based practices into real-world settings (Eccles and Mittman, Implement Sci. 1(1):1, 2006). Central to the identification of replicable, feasible, and effective implementation strategies is the ability to assess the impact of contextual constructs and intervention characteristics that may influence implementation, but several measurement issues make this work quite difficult. For instance, it is unclear which constructs have no measures and which measures have any evidence of psychometric properties like reliability and validity. As part of a larger set of studies to advance implementation science measurement (Lewis et al., Implement Sci. 10:102, 2015), we will complete systematic reviews of measures that map onto the Consolidated Framework for Implementation Research (Damschroder et al., Implement Sci. 4:50, 2009) and the Implementation Outcomes Framework (Proctor et al., Adm Policy Ment Health. 38(2):65-76, 2011), the protocol for which is described in this manuscript. METHODS: Our primary databases will be PubMed and Embase. Our search strings will be comprised of five levels: (1) the outcome or construct term; (2) terms for measure; (3) terms for evidence-based practice; (4) terms for implementation; and (5) terms for mental health. Two trained research specialists will independently review all titles and abstracts followed by full-text review for inclusion. The research specialists will then conduct measure-forward searches using the "cited by" function to identify all published empirical studies using each measure. The measure and associated publications will be compiled in a packet for data extraction. Data relevant to our Psychometric and Pragmatic Evidence Rating Scale (PAPERS) will be independently extracted and then rated using a worst score counts methodology reflecting "poor" to "excellent" evidence. DISCUSSION: We will build a centralized, accessible, searchable repository through which researchers, practitioners, and other stakeholders can identify psychometrically and pragmatically strong measures of implementation contexts, processes, and outcomes. By facilitating the employment of psychometrically and pragmatically strong measures identified through this systematic review, the repository would enhance the cumulativeness, reproducibility, and applicability of research findings in the rapidly growing field of implementation science.