Research outcomes informing the selection of public health interventions and strategies to implement them: A cross-sectional survey of Australian policy-maker and practitioner preferences.

BACKGROUND: A key role of public health policy-makers and practitioners is to ensure beneficial interventions are implemented effectively enough to yield improvements in public health. The use of evidence to guide public health decision-making to achieve this is recommended. However, few studies have examined the relative value, as reported by policy-makers and practitioners, of different broad research outcomes (that is, measures of cost, acceptability, and effectiveness). To guide the conduct of research and better inform public health policy and practice, this study aimed at describing the research outcomes that Australian policy-makers and practitioners consider important for their decision-making when selecting: (a) public health interventions; (b) strategies to support their implementation; and (c) to assess the differences in research outcome preferences between policy-makers and practitioners. METHOD: An online value-weighting survey was conducted with Australian public health policy-makers and practitioners working in the field of non-communicable disease prevention. Participants were presented with a list of research outcomes and were asked to select up to five they considered most critical to their decision-making. They then allocated 100 points across these - allocating more points to outcomes perceived as more important. Outcome lists were derived from a review and consolidation of evaluation and outcome frameworks in the fields of public health knowledge translation and implementation. We used descriptive statistics to report relative preferences overall and for policy-makers and practitioners separately. RESULTS: Of the 186 participants; 90 primarily identified as policy-makers and 96 as public health prevention practitioners. Overall, research outcomes of effectiveness, equity, feasibility, and sustainability were identified as the four most important outcomes when considering either interventions or strategies to implement them. Scores were similar for most outcomes between policy-makers and practitioners. CONCLUSION: For Australian policy-makers and practitioners working in the field of non-communicable disease prevention, outcomes related to effectiveness, equity, feasibility, and sustainability appear particularly important to their decisions about the interventions they select and the strategies they employ to implement them. The findings suggest researchers should seek to meet these information needs and prioritize the inclusion of such outcomes in their research and dissemination activities. The extent to which these outcomes are critical to informing the decision of policy-makers and practitioners working in other jurisdictions or contexts warrants further investigation.

Disseminating health research to public health policy-makers and practitioners: a survey of source, message content and delivery modality preferences.

BACKGROUND: Understanding the views of policy-makers and practitioners regarding how best to communicate research evidence is important to support research use in their decision-making. AIM: To quantify and describe public health policy-makers and practitioners' views regarding the source, content and form of messages describing public health research findings to inform their decision-making. We also sought to examine differences in preferences between public health policy-makers and practitioners. METHODS: A cross sectional, value-weighting survey of policy-makers and practitioners was conducted. Participants were asked to allocate a proportion of 100 points across different (i) sources of research evidence, (ii) message content and (iii) the form in which evidence is presented. Points were allocated based on their rating of influence, usefulness and preference when making decisions about health policy or practice. RESULTS: A total of 186 survey responses were received from 90 policy-makers and 96 practitioners. Researchers and government department agencies were the most influential source of research evidence based on mean allocation of points, followed by knowledge brokers, professional peers and associations. Mean point allocation for perceived usefulness of message content was highest for simple summary of key findings and implications, and then evidence-based recommendations and data and statistical summaries. Finally, based on mean scores, policy-makers and practitioners preferred to receive research evidence in the form of peer-reviewed publications, reports, evidence briefs and plain language summaries. There were few differences in scores between policy-makers and practitioners across source, message content or form assessments or those with experience in different behavioural areas. CONCLUSIONS: The findings should provide a basis for the future development and optimization of dissemination strategies to this important stakeholder group.

Physical Activity and Depression and Anxiety Disorders: A Systematic Review of Reviews and Assessment of Causality.

Introduction: Globally, depressive and anxiety disorders are the leading contributors to mental ill health. Physical activity reduces symptoms of depression and anxiety and has been proposed as an adjunct treatment therapy for depression and anxiety. Prospective studies suggest that physical activity may reduce the incidence of depression and anxiety. We conducted a systematic review of reviews with the aim to provide a comprehensive overview of available epidemiologic evidence on the strength of the association between physical activity and incident cases of depression and anxiety and to assess the likelihood of these associations being causal. Methods: We searched Embase and PubMed databases for systematic reviews published between January 1, 2000 and March 19, 2020 that reported findings on the strength of association between physical activity and incidence of depression and anxiety. We updated this search to October 15, 2022. Two reviewers independently assessed the methodologic quality of the included reviews using the Assessment of Multiple Systematic Reviews rating scale. We carried out a narrative synthesis of the evidence. We used the Bradford Hill criteria to assess the likelihood of associations being causal. Results: The initial search yielded 770 articles, of which 4 remained for data extraction. Two of the included reviews were scored as high quality, and 2 were scored as low quality. From the 2 included reviews that reported pooled estimates, people with high physical activity levels were found to have a decreased risk of incident depression (adjusted RR=0.83, 95% CI=0.76, 0.90) and reduced odds of developing anxiety (adjusted OR=0.74,95% CI=0.62, 0.88) when compared with those with low physical activity levels. We assessed physical activity to be probably causally related to both depression and anxiety. Discussion: Our evidence is drawn from systematic reviews of observational data. Further high-quality studies, such as randomized control trials, would help to strengthen the evidence base of the associations between physical activity and depression and anxiety. Nonetheless, our findings provide empirical support for the consideration of physical activity in strategies for the prevention of mental ill health.

Study protocol of an economic evaluation embedded in the Midwives and Obstetricians Helping Mothers to Quit Smoking (MOHMQuit) trial.

BACKGROUND: Tobacco smoking during pregnancy is the most important preventable risk factor for pregnancy complications and adverse birth outcomes and can have lifelong consequences for infants. Smoking during pregnancy is associated with higher healthcare costs related to birth complications and during childhood. Psychosocial interventions to support pregnant women to quit are effective, yet provision of smoking cessation support has been inconsistent. The Midwives and Obstetricians Helping Mothers to Quit Smoking (MOHMQuit) intervention provides systems change, and leadership and clinician elements, to support clinicians to help women stop smoking in pregnancy. There have been few long-term analyses conducted of the cost-effectiveness of smoking cessation interventions for pregnant women that target healthcare providers. This protocol describes the economic evaluation of the MOHMQuit trial, a pragmatic stepped-wedge cluster-randomised controlled implementation trial in nine public maternity services in New South Wales (NSW), Australia, to ascertain whether MOHMQuit is cost-effective in supporting clinicians to help women quit smoking in pregnancy compared to usual care. METHODS: Two primary analyses will be carried out comparing MOHMQuit with usual care from an Australian health care system perspective: i) a within-trial cost-effectiveness analysis with results presented as the incremental cost per additional quitter; and ii) a lifetime cost-utility analysis using a published probabilistic decision analytic Markov model with results presented as incremental cost per quality-adjusted life-year (QALY) gained for mother and child. Patient-level data on resource use and outcomes will be used in the within-trial analysis and extrapolated and supplemented with national population statistics and published data from the literature for the lifetime analysis. DISCUSSION: There is increasing demand for information on the cost-effectiveness of implementing healthcare interventions to provide policy makers with critical information for the best value for money within finite budgets. Economic evaluation of the MOHMQuit trial will provide essential, policy-relevant information for decision makers on the value of evidence-based implementation of support for healthcare providers delivering services for pregnant women. TRIAL REGISTRATIONS: ACTRN12622000167763, registered 2 February 2022.

Evidence on Scaling in Health and Social Care: An Umbrella Review.

Policy Points More rigorous methodologies and systematic approaches should be encouraged in the science of scaling. This will help researchers better determine the effectiveness of scaling, guide stakeholders in the scaling process, and ultimately increase the impacts of health innovations. The practice and the science of scaling need to expand worldwide to address complex health conditions such as noncommunicable and chronic diseases. Although most of the scaling experiences described in the literature are occurring in the Global South, most of the authors publishing on it are based in the Global North. As the science of scaling spreads across the world with the aim of reducing health inequities, it is also essential to address the power imbalance in how we do scaling research globally. CONTEXT: Scaling of effective innovations in health and social care is essential to increase their impact. We aimed to synthesize the evidence base on scaling and identify current knowledge gaps. METHODS: We conducted an umbrella review according to the Joanna Briggs Institute Reviewers' Manual. We included any type of review that 1) focused on scaling, 2) covered health or social care, and 3) presented a methods section. We searched MEDLINE (Ovid), Embase, PsycINFO (Ovid), CINAHL (EBSCO), Web of Science, The Cochrane Library, Sociological Abstracts (ProQuest), Academic Search Premier (EBSCO), and ProQuest Dissertations & Theses Global from their inception to August 6, 2020. We searched the gray literature using, e.g., Google and WHO-ExpandNet. We assessed methodological quality with AMSTAR2. Paired reviewers independently selected and extracted eligible reviews and assessed study quality. A narrative synthesis was performed. FINDINGS: Of 24,269 records, 137 unique reviews were included. The quality of the 58 systematic reviews was critically low (n = 42). The most frequent review type was systematic review (n = 58). Most reported on scaling in low- and middle-income countries (n = 59), whereas most first authors were from high-income countries (n = 114). Most reviews concerned infectious diseases (n = 36) or maternal-child health (n = 28). They mainly focused on interventions (n = 37), barriers and facilitators (n = 29), frameworks (n = 24), scalability (n = 24), and costs (n = 14). The WHO/ExpandNet scaling definition was the definition most frequently used (n = 26). Domains most reported as influencing scaling success were building scaling infrastructure (e.g., creating new service sites) and human resources (e.g., training community health care providers). CONCLUSIONS: The evidence base on scaling is evolving rapidly as reflected by publication trends, the range of focus areas, and diversity of scaling definitions. Our study highlights knowledge gaps around methodology and research infrastructures to facilitate equitable North-South research relationships. Common efforts are needed to ensure scaling expands the impacts of health and social innovations to broader populations.

Tensions and Paradoxes of Scaling Up: A Critical Reflection on Physical Activity Promotion.

BACKGROUND: Achieving system-level, sustainable 'scale-up' of interventions is the epitome of successful translation of evidence-based approaches in population health. In physical activity promotion, few evidence-based interventions reach implementation at scale or become embedded within systems for sustainable health impact. This is despite the vast published literature describing efficacy studies of small-scale physical activity interventions. Research into physical activity scale-up (through case-study analysis; evaluations of scale-up processes in implementation trials; and mapping the processes, strategies, and principles for scale-up) has identified barriers and facilitators to intervention expansion. Many interventions are implemented at scale by governments but have not been evaluated or have unpublished evaluation information. Further, few public health interventions have evaluations that reveal the costs and benefits of scaled-up implementation. This lack of economic information introduces an additional element of risk for decision makers when deciding which physical activity interventions should be supported with scarce funding resources. Decision-makers face many other challenges when scaling interventions which do not relate to formal research trials of scale-up; Methods: To explore these issues, a multidisciplinary two-day workshop involving experts in physical activity scale-up was convened by the University of Newcastle, Australia, and the University of Ottawa, Canada (February 2019); Results: In this paper we discuss some of the scale-up tensions (challenges and conflicts) and paradoxes (things that are contrary to expectations) that emerged from this workshop in the context of the current literature and our own experiences in this field. We frame scale-up tensions according to epistemology, methodology, time, and partnerships; and paradoxes as 'reach without scale', 'planned serendipity' and 'simple complexity'. We reflect on the implications of these scale-up tensions and paradoxes, providing considerations for future scale-up research and practice moving forward; Conclusions: In this paper, we delve deeper into stakeholders' assumptions, processes and expectations of scaling up, and challenge in what ways as stakeholders, we all contribute to desired or undesired outcomes. Through a lens of 'tensions' and 'paradoxes', we make an original contribution to the scale-up literature that might influence current perspectives of scaling-up, provide future approaches for physical activity promotion, and contribute to understanding of dynamic of research-practice partnerships.

A cost-benefit analysis framework for preventive health interventions to aid decision-making in Australian governments.

BACKGROUND: Australian governments are increasingly mandating the use of cost-benefit analysis (CBA) to inform the efficient allocation of government resources. CBA is likely to be useful when evaluating preventive health interventions that are often cross-sectoral in nature and require Cabinet approval prior to implementation. This study outlines a CBA framework for the evaluation of preventive health interventions that balances the need for consistency with other agency guidelines whilst adhering to guidelines and conventions for health economic evaluations. METHODS: We analysed CBA and other evaluation guidance documents published by Australian federal and New South Wales (NSW) government departments. Data extraction compared the recommendations made by different agencies and the impact on the analysis of preventive health interventions. The framework specifies a reference case and sensitivity analyses based on the following considerations: (1) applied economic evaluation theory; (2) consistency between CBA across different government departments; (3) the ease of moving from a CBA to a more conventional cost-effectiveness/cost-utility analysis framework often used for health interventions; (4) the practicalities of application; and (5) the needs of end users being both Cabinet decision-makers and health policy-makers. RESULTS: Nine documents provided CBA or relevant economic evaluation guidance. There were differences in terminology and areas of agreement and disagreement between the guidelines. Disagreement between guidelines involved (1) the community included in the societal perspective; (2) the number of options that should be appraised in ex ante analyses; (3) the appropriate time horizon for interventions with longer economic lives; (4) the theoretical basis and value of the discount rate; (5) parameter values for variables such as the value of a statistical life; and (6) the summary measure for decision-making. CONCLUSIONS: This paper addresses some of the methodological challenges that have hindered the use of CBA in prevention by outlining a framework that is consistent with treasury department guidelines whilst considering the unique features of prevention policies. The effective use and implementation of a preventive health CBA framework is likely to require considerable investment of time and resources from state and federal government departments of health and treasury but has the potential to improve decision-making related to preventive health policies and programmes.

Evidence on scaling in health and social care: protocol for a living umbrella review.

BACKGROUND: There is a growing interest in scaling effective health innovations to promote equitable access to high-quality health services worldwide. However, multiple challenges persist in scaling innovations. In this study, we aim to summarize the scaling evidence in the health and social care literature and identify current knowledge gaps. METHODS: We will conduct a living umbrella review according to the Joanna Briggs Institute Reviewers' Manual. We will consider all knowledge syntheses addressing scaling in health or social care (e.g., any setting, any clinical area) and conducted in a systematic way. We will search the following electronic databases: MEDLINE (Ovid), Embase, PsychINFO (Ovid), CINAHL (EBSCO), Web of Science, The Cochrane Library, Sociological Abstract (Proquest), Academic Search Premier (EBSCO), and Proquest Dissertations & Theses Global, from inception. Furthermore, we will conduct searches of the grey literature. No restriction regarding date or language will be applied. Each phase of the review will be processed by two independent reviewers. We will develop a data extraction form on Covidence. We will assess the methodological quality of the included reviews using AMSTAR2 and the risk of bias using ROBIS. Results will be presented in tabular form and accompanied by a narrative synthesis covering the traditional themes of scaling science that emerge from the analysis, such as coverage, range, and sustainability, as well as themes less covered in the literature, including reporting guidance, models, tools, barriers, and/or facilitators to scaling innovations, evidence regarding application in high-income or low-income countries, and end-user engagement. We will disseminate the findings via publications and through relevant networks. DISCUSSION: The findings of the umbrella review will facilitate access to scaling evidence in the literature and help strengthen the science of scaling for researchers, policy makers, and program managers. Finally, this work will highlight important knowledge gaps and help prioritize future research questions. SYSTEMATIC REVIEW REGISTRATION: This protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on November 11, 2020 (registration number: CRD42020183774 ).

"Getting Ireland Active"-Application of a Systems Approach to Increase Physical Activity in Ireland Using the GAPPA Framework.

Physical activity (PA) promotion is a complex challenge, with the Global Action Plan on Physical Activity (GAPPA) endorsing a systems approach and recommending countries assess existing areas of progress which can be strengthened. This paper reports a process facilitating a systems approach for identifying current good practice and gaps for promoting PA in Ireland. Elements of participatory action research were enabled through 3 stages: (1) aligning examples of actions from Irish policy documents (n = 3) to the GAPPA, (2) workshop with stakeholders across multiple sectors, and (3) review of outputs. Data collected through the workshop were analyzed using a deductive thematic analysis guided by the GAPPA. The policy context in Ireland aligns closely to the GAPPA with the creation of Active Systems the most common strategic objective across policy documents. Forty participants (50% male) took part in the systems approach workshop, which after revision resulted in 80 examples of good practice and 121 actions for greater impact. A pragmatic and replicable process facilitating a systems approach was adopted and showed current Irish policy and practices align with the GAPPA "good practices." The process provides existing areas of progress which can be strengthened, as well as the policy opportunities and practice gaps.

Systematic Review to Update 'Value of a Statistical Life' Estimates for Australia.

The value of a statistical life (VSL) estimates individuals' willingness to trade wealth for mortality risk reduction. This economic parameter is often a major component of the quantified benefits estimated in the evaluation of government policies related to health and safety. This study reviewed the literature to update the VSL recommended for Australian policy appraisals. A systematic literature review was conducted to capture Australian primary studies and international review papers reporting VSL estimates published from 2007 to January 2019. International estimates were adjusted for income differences and the median VSL estimate was extracted from each review study. VSL estimates were used to calculate the value of a statistical life year. Of the 18 studies that met the inclusion criteria, two studies were primary Australian studies with a weighted mean VSL of A$7.0 million in 2017 values. The median VSL in the review studies was A$7.3 million. For Australian public policy appraisals, we recommend the consideration of a base case VSL for people of all ages and across all risk contexts of A$7.0 million. Sensitivity analyses could use a high value of A$7.3 million and a low value that reflects the value (A$4.3 million) currently recommended by the Australian government.

Strategies for involving patients and the public in scaling-up initiatives in health and social services: protocol for a scoping review and Delphi survey.

BACKGROUND: The scale-up of evidence-based innovations is required to reduce waste and inequities in health and social services (HSS). However, it often tends to be a top-down process initiated by policy makers, and the values of the intended beneficiaries are forgotten. Involving multiple stakeholders including patients and the public in the scaling-up process is thus essential but highly complex. We propose to identify relevant strategies for meaningfully and equitably involving patients and the public in the science and practice of scaling up in HSS. METHODS: We will adapt our overall method from the RAND/UCLA Appropriateness Method. Following this, we will perform a two-prong study design (knowledge synthesis and Delphi study) grounded in an integrated knowledge translation approach. This approach involves extensive participation of a network of stakeholders interested in patient and public involvement (PPI) in scaling up and a multidisciplinary steering committee. We will conduct a systematic scoping review following the methodology recommended in the Joanna Briggs Institute Reviewers Manual. We will use the following eligibility criteria: (1) participants-any stakeholder involved in creating or testing a strategy for PPI; (2) intervention-any PPI strategy proposed for scaling-up initiatives; (3) comparator-no restriction; (4) outcomes: any process or outcome metrics related to PPI; and (5) setting-HSS. We will search electronic databases (e.g., Medline, Web of Science, Sociological Abstract) from inception onwards, hand search relevant websites, screen the reference lists of included records, and consult experts in the field. Two reviewers will independently select and extract eligible studies. We will summarize data quantitatively and qualitatively and report results using the PRISMA extension for Scoping Reviews (PRISMA-ScR) checklist. We will conduct an online Delphi survey to achieve consensus on the relevant strategies for PPI in scaling-up initiatives in HSS. Participants will include stakeholders from low-, middle-, and high-income countries. We anticipate that three rounds will allow an acceptable degree of agreement on research priorities. DISCUSSION: Our findings will advance understanding of how to meaningfully and equitably involve patients and the public in scaling-up initiatives for sustainable HSS. SYSTEMATIC REVIEW REGISTRATION: We registered this protocol with the Open Science Framework on August 19, 2020 ( https://osf.io/zqpx7/ ).

Physical activity coaching for adults with mobility limitations: protocol for the ComeBACK pragmatic hybrid effectiveness-implementation type 1 randomised controlled trial.

INTRODUCTION: Mobility limitation is common and often results from neurological and musculoskeletal health conditions, ageing and/or physical inactivity. In consultation with consumers, clinicians and policymakers, we have developed two affordable and scalable intervention packages designed to enhance physical activity for adults with self-reported mobility limitations. Both are based on behaviour change theories and involve tailored advice from physiotherapists. METHODS AND ANALYSIS: This pragmatic hybrid effectiveness-implementation type 1 randomised control trial (n=600) will be undertaken among adults with self-reported mobility limitations. It aims to estimate the effects on physical activity of: (1) an enhanced 6-month intervention package (one face-to-face physiotherapy assessment, tailored physical activity plan, physical activity phone coaching from a physiotherapist, informational/motivational resources and activity monitors) compared with a less intensive 6-month intervention package (single session of tailored phone advice from a physiotherapist, tailored physical activity plan, unidirectional text messages, informational/motivational resources); (2) the enhanced intervention package compared with no intervention (6-month waiting list control group); and (3) the less intensive intervention package compared with no intervention (waiting list control group). The primary outcome will be average steps per day, measured with the StepWatch Activity Monitor over a 1-week period, 6 months after randomisation. Secondary outcomes include other physical activity measures, measures of health and functioning, individualised mobility goal attainment, mental well-being, quality of life, rate of falls, health utilisation and intervention evaluation. The hybrid effectiveness-implementation design (type 1) will be used to enable the collection of secondary implementation outcomes at the same time as the primary effectiveness outcome. An economic analysis will estimate the cost-effectiveness and cost-utility of the interventions compared with no intervention and to each other. ETHICS AND DISSEMINATION: Ethical approval has been obtained by Sydney Local Health District, Royal Prince Alfred Zone. Dissemination will be via publications, conferences, newsletters, talks and meetings with health managers. TRIAL REGISTRATION NUMBER: ACTRN12618001983291.

The Intervention Scalability Assessment Tool: a pilot study assessing five interventions for scalability.

OBJECTIVES: This study tested the utility of the Intervention Scalability Assessment Tool (ISAT) with real-world interventions considered for scale-up in the Australian context. We explored user perceptions of the usefulness of the ISAT in making scalability decisions and conducted an analysis of completed ISATs. Type of program/service: The ISAT was designed to assist policy makers/practitioners to make structured and reflective assessments of the potential scalability of their interventions. METHODS: Participants (n = 26) were invited via email and needed to be considering a population intervention for scale-up, to be able to pilot the ISAT, and to be willing to complete an online survey to provide feedback. Survey data and completed ISATs were collated and common responses were identified, with exceptions noted. RESULTS: Thirteen participants accepted the invitation to pilot the ISAT; six completed ISATs and returned surveys and five of each were included in the final analysis. The process by which participants completed the ISAT varied, some involving up to six stakeholders in the process and others only one. The time taken to complete the ISAT ranged from 2 hours to 3 days. Interventions considered for scalability were public health interventions and all but one involved a potential statewide scale-up. In terms of outcome, scalability scores across interventions varied, however, all but one intervention was recommended for scale-up. The remaining intervention was deemed to be promising but further information was required before assessing scalability. LESSONS LEARNT: The ISAT was perceived as a useful tool to assess the scalability of real-world health programs, with participants noting only minor limitations and challenges. The ISAT can be applied to a variety of population health interventions and identifies strengths and weaknesses in the evidence base for scale-up. This study revealed that decisions to scale up are often made despite potential weaknesses highlighted through the ISAT. This is likely a reflection of the critically influential role of contextual factors in decisions to scale up, whereby decision makers may balance a variety of considerations, of which evidence of scalability is only one factor.

Can an electronic monitoring system capture implementation of health promotion programs? A focussed ethnographic exploration of the story behind program monitoring data.

BACKGROUND: There is a pressing need for policy makers to demonstrate progress made on investments in prevention, but few examples of monitoring systems capable of tracking population-level prevention policies and programs and their implementation. In New South Wales, Australia, the scale up of childhood obesity prevention programs to over 6000 childcare centres and primary schools is monitored via an electronic monitoring system, "PHIMS". METHODS: Via a focussed ethnography with all 14 health promotion implementation teams in the state, we set out to explore what aspects of program implementation are captured via PHIMS, what aspects are not, and the implications for future IT implementation monitoring systems as a result. RESULTS: Practitioners perform a range of activities in the context of delivering obesity prevention programs, but only specific activities are captured via PHIMS. PHIMS thereby defines and standardises certain activities, while non-captured activities can be considered as "extra" work by practitioners. The achievement of implementation targets is influenced by multi-level contextual factors, with only some of the factors accounted for in PHIMS. This evidences incongruencies between work done, recorded and, therefore, recognised. CONCLUSIONS: While monitoring systems cannot and should not capture every aspect of implementation, better accounting for aspects of context and "extra" work involved in program implementation could help illuminate why implementation succeeds or fails. Failure to do so may result in policy makers drawing false conclusions about what is required to achieve implementation targets. Practitioners, as experts of context, are well placed to assist policy makers to develop accurate and meaningful implementation targets and approaches to monitoring.

Intervention Scalability Assessment Tool: A decision support tool for health policy makers and implementers.

BACKGROUND: Promising health interventions tested in pilot studies will only achieve population-wide impact if they are implemented at scale across communities and health systems. Scaling up effective health interventions is vital as not doing so denies the community the most effective services and programmes. However, there remains a paucity of practical tools to assess the suitability of health interventions for scale-up. The Intervention Scalability Assessment Tool (ISAT) was developed to support policy-makers and practitioners to make systematic assessments of the suitability of health interventions for scale-up. METHODS: The ISAT was developed over three stages; the first stage involved a literature review to identify similar tools and frameworks that could be used to guide scalability assessments, and expert input to develop draft ISAT content. In the second stage, the draft ISAT tool was tested with end users. The third stage involved revising and re-testing the ISAT with end users to further refine the language and structure of the final ISAT. RESULTS: A variety of information and sources of evidence should be used to complete the ISAT. The ISAT consists of three parts. Part A: 'setting the scene' requires consideration of the context in which the intervention is being considered for scale-up and consists of five domains, as follows: (1) the problem; (2) the intervention; (3) strategic/political context; (4) evidence of effectiveness; and (5) intervention costs and benefits. Part B asks users to assess the potential implementation and scale-up requirements within five domains, namely (1) fidelity and adaptation; (2) reach and acceptability; (3) delivery setting and workforce; (4) implementation infrastructure; and (5) sustainability. Part C generates a graphical representation of the strengths and weaknesses of the readiness of the proposed intervention for scale-up. Users are also prompted for a recommendation as to whether the intervention (1) is recommended for scale-up, (2) is promising but needs further information before scaling up, or (3) does not yet merit scale-up. CONCLUSION: The ISAT fills an important gap in applied scalability assessment and can become a critical decision support tool for policy-makers and practitioners when selecting health interventions for scale-up. Although the ISAT is designed to be a health policy and practitioner tool, it can also be used by researchers in the design of research to fill important evidence gaps.

Implementation and scale-up of physical activity and behavioural nutrition interventions: an evaluation roadmap.

BACKGROUND: Interventions that work must be effectively delivered at scale to achieve population level benefits. Researchers must choose among a vast array of implementation frameworks (> 60) that guide design and evaluation of implementation and scale-up processes. Therefore, we sought to recommend conceptual frameworks that can be used to design, inform, and evaluate implementation of physical activity (PA) and nutrition interventions at different stages of the program life cycle. We also sought to recommend a minimum data set of implementation outcome and determinant variables (indicators) as well as measures and tools deemed most relevant for PA and nutrition researchers. METHODS: We adopted a five-round modified Delphi methodology. For rounds 1, 2, and 3 we administered online surveys to PA and nutrition implementation scientists to generate a rank order list of most commonly used; i) implementation and scale-up frameworks, ii) implementation indicators, and iii) implementation and scale-up measures and tools. Measures and tools were excluded after round 2 as input from participants was very limited. For rounds 4 and 5, we conducted two in-person meetings with an expert group to create a shortlist of implementation and scale-up frameworks, identify a minimum data set of indicators and to discuss application and relevance of frameworks and indicators to the field of PA and nutrition. RESULTS: The two most commonly referenced implementation frameworks were the Framework for Effective Implementation and the Consolidated Framework for Implementation Research. We provide the 25 most highly ranked implementation indicators reported by those who participated in rounds 1-3 of the survey. From these, the expert group created a recommended minimum data set of implementation determinants (n = 10) and implementation outcomes (n = 5) and reconciled differences in commonly used terms and definitions. CONCLUSIONS: Researchers are confronted with myriad options when conducting implementation and scale-up evaluations. Thus, we identified and prioritized a list of frameworks and a minimum data set of indicators that have potential to improve the quality and consistency of evaluating implementation and scale-up of PA and nutrition interventions. Advancing our science is predicated upon increased efforts to develop a common 'language' and adaptable measures and tools.

Capturing implementation knowledge: applying focused ethnography to study how implementers generate and manage knowledge in the scale-up of obesity prevention programs.

BACKGROUND: Bespoke electronic information management systems are being used for large-scale implementation delivery of population health programs. They record sites reached, coordinate activity, and track target achievement. However, many systems have been abandoned or failed to integrate into practice. We investigated the unusual endurance of an electronic information management system that has supported the successful statewide implementation of two evidence-based childhood obesity prevention programs for over 5 years. Upwards of 80% of implementation targets are being achieved. METHODS: We undertook co-designed partnership research with policymakers, practitioners, and IT designers. Our working hypothesis was that the science of getting evidence-based programs into practice rests on an in-depth understanding of the role programs play in the ongoing system of local relationships and multiple accountabilities. We conducted a 12-month multisite ethnography of 14 implementation teams, including their use of an electronic information management system, the Population Health Information Management System (PHIMS). RESULTS: All teams used PHIMS, but also drew on additional informal tools and technologies to manage, curate, and store critical information for implementation. We identified six functions these tools performed: (1) relationship management, (2) monitoring progress towards target achievement, (3) guiding and troubleshooting PHIMS use, (4) supporting teamwork, (5) evaluation, and (6) recording extra work at sites not related to program implementation. Informal tools enabled practitioners to create locally derived implementation knowledge and provided a conduit between knowledge generation and entry into PHIMS. CONCLUSIONS: Implementation involves knowing and formalizing what to do, as well as how to do it. Our ethnography revealed the importance of hitherto uncharted knowledge about how practitioners develop implementation knowledge about how to do implementation locally, within the context of scaling up. Harnessing this knowledge for local use required adaptive and flexible systems which were enabled by informal tools and technologies. The use of informal tools also complemented and supported PHIMS use suggesting that both informal and standardized systems are required to support coordinated, large-scale implementation. While the content of the supplementary knowledge required to deliver the program was specific to context, functions like managing relationships with sites and helping others in the team may be applicable elsewhere.

Can a continuous quality improvement program create culturally safe emergency departments for Aboriginal people in Australia? A multiple baseline study.

BACKGROUND: Providing culturally safe health care can contribute to improved health among Aboriginal people. However, little is known about how to make hospitals culturally safe for Aboriginal people. This study assessed the impact of an emergency department (ED)-based continuous quality improvement program on: the accuracy of recording of Aboriginal status in ED information systems; incomplete ED visits among Aboriginal patients; and the cultural appropriateness of ED systems and environments. METHODS: Between 2012 and 2014, the Aboriginal Identification in Hospitals Quality Improvement Program (AIHQIP) was implemented in eight EDs in NSW, Australia. A multiple baseline design and analysis of linked administrative data were used to assess program impact on the proportion of Aboriginal patients correctly identified as Aboriginal in ED information systems and incomplete ED visits in Aboriginal patients. Key informant interviews and document review were used to explore organisational changes. RESULTS: In all EDs combined, the AIHQIP was not associated with a reduction in incomplete ED visits in Aboriginal people, nor did it influence the proportion of ED visits made by Aboriginal people that had an accurate recording of Aboriginal status. However, in two EDs it was associated with an increase in the trend of accurate recording of Aboriginality from baseline to the intervention period (odds ratio (OR) 1.31, p < 0.001 in ED 4 and OR 1.15, p = 0.020 in ED 5). In other words, the accuracy of recording of Aboriginality increased from 61.4 to 70% in ED 4 and from 72.6 to 73.9% in ED 5. If the program were not implemented, only a marginal increase would have occurred in ED 4 (from 61.4 to 64%) and, in ED 5, the accuracy of reporting would have decreased (from 72.6 to 71.1%). Organisational changes were achieved across EDs, including modifications to waiting areas and improved processes for identifying Aboriginal patients and managing incomplete visits. CONCLUSIONS: The AIHQIP did not have an overall effect on the accuracy of recording of Aboriginal status or on levels of incomplete ED visits in Aboriginal patients. However, important organisational changes were achieved. Further research investigating the effectiveness of interventions to improve Aboriginal cultural safety is warranted.

Barriers to the evaluation of evidence-based public health policy.

Public health policy has the potential to produce great benefits for individuals and communities. There is growing demand that such efforts be rigorously evaluated to ensure that the expected benefits are, in fact, realised. Commonly, public health policy is evaluated by consumer acceptability, reach, or changes in knowledge and attitudes. Non-robust research designs are often used. But these approaches to evaluation do not answer three critical questions: Has a change in the desired outcome occurred? Was it a consequence of the policy and not some extraneous factor? Was the size of the change considered significant and cost-effective? We, a team of government and academic scholars working in research and evaluation, have examined some of the more common impediments to robust evaluation: political impediments, a lack of investment in evaluation capacity within bureaucracy, and the failure of academic researchers to understand the need for the evaluation of public health policy.

Process evaluation of the midwifery initiated oral health-dental service program: Perceptions of dental professionals.

ISSUE ADDRESSED: All pregnant women should have a comprehensive oral health evaluation. Unfortunately, many pregnant women seldom seek dental care and some dentists are hesitant to treat during pregnancy. To address these issues, the Midwifery Initiated Oral Health Dental Service (MIOH-DS) program was developed in Australia. The aim of this study was to undertake a process evaluation and explore the perceptions of dental professionals involved in the program to determine the acceptability, feasibility and effectiveness of the program if it were to be upscaled. METHODS: A qualitative approach using content analysis was conducted on data from two focus groups involving 12 dental professionals. RESULTS: All participants were supportive of the MIOH-DS program. They thought pregnant women were receptive to their care, and reported markedly improved oral health. The provision of free dental care and the involvement of midwives were cited as major factors that improved the uptake of the program. Some of the challenges encountered were the prevailing misconceptions about the safety of dental treatment and pregnancy-related impairments. CONCLUSIONS: Dental professionals found the MIOH-DS to be acceptable, feasible and effective in improving oral health of pregnant women and their uptake of dental services. However, some challenges need to be addressed as the MIOH-DS program is upscaled into a cost-effective model. SO WHAT?: Dental professionals are important stakeholders in the MIOH-DS model. The process evaluation of the successful dental intervention is necessary to understand how and why such interventions work, and is an important step in scaling up to a population-wide intervention.