Increasing the capacity of policy agencies to use research findings: a stepped-wedge trial.

BACKGROUND: This paper describes the trial of a novel intervention, Supporting Policy In health with evidence from Research: an Intervention Trial (SPIRIT). It examines (1) the feasibility of delivering this kind of programme in practice; (2) its acceptability to participants; (3) the impact of the programme on the capacity of policy agencies to engage with research; and (4) the engagement with and use of research by policy agencies. METHODS: SPIRIT was a multifaceted, highly tailored, stepped-wedge, cluster-randomised, trial involving six health policy agencies in Sydney, Australia. Agencies were randomly allocated to one of three start dates to receive the 1-year intervention programme. SPIRIT included audit, feedback and goal setting; a leadership programme; staff training; the opportunity to test systems to facilitate research use in policies; and exchange with researchers. Outcome measures were collected at each agency every 6 months for 30 months. RESULTS: Participation in SPIRIT was associated with significant increases in research use capacity at staff and agency levels. Staff reported increased confidence in research use skills, and agency leaders reported more extensive systems and structures in place to support research use. Self-report data suggested there was also an increase in tactical research use among agency staff. Given the relatively small numbers of participating agencies and the complexity of their contexts, findings suggest it is possible to effect change in the way policy agencies approach the use of research. This is supported by the responses on the other trial measures; while these were not statistically significant, on 18 of the 20 different measures used, the changes observed were consistent with the hypothesised intervention effect (that is, positive impacts). CONCLUSIONS: As an early test of an innovative approach, SPIRIT has demonstrated that it is possible to increase research engagement and use in policy agencies. While more work is needed to establish the replicability and generalisability of these findings, this trial suggests that building staff skills and organisational structures may be effective in increasing evidence use.

Do policy-makers find commissioned rapid reviews useful?

BACKGROUND: Rapid reviews are increasingly used by policy agencies to access relevant research in short timeframes. Despite the growing number of programmes, little is known about how rapid reviews are used by health policy agencies. This study examined whether and how rapid reviews commissioned using a knowledge brokering programme were used by Australian policy-makers. METHODS: This study used interview data to examine the use of 139 rapid reviews by health policy agencies that were commissioned between 2006 and 2015. Transcripts were coded to identify how rapid reviews were used, the type of policy processes in which they were used, what evidence of use was provided and what reasons were given when rapid reviews were not used. Fisher's exact test was used to assess variation between types of agencies. RESULTS: Overall, 89% of commissioned rapid reviews were used by the commissioning agencies and 338 separate instances of use were identified, namely, on average, three uses per review. Policy-makers used reviews primarily to determine the details of a policy or programme, identify priorities for future action or investment, negotiate interjurisdictional decisions, evaluate alternative solutions for a policy problem, and communicate information to stakeholders. Some variation in use was observed across agencies. Reasons for non-use were related to changes in organisational structures, resources or key personnel in the commissioning agencies, or changes in the broader political environment. CONCLUSIONS: This study found that almost all rapid reviews had been used by the agencies who commissioned them, primarily in policy and programme development, agenda-setting, and to communicate information to stakeholders. Reviews were used mostly in instrumental and conceptual ways and there was little evidence of symbolic use. Variations in use were identified across agencies. The findings suggest that commissioned rapid reviews are an effective means of providing timely relevant research for use in policy processes and that review findings may be applied in a variety of ways.

What can we learn from interventions that aim to increase policy-makers' capacity to use research? A realist scoping review.

BACKGROUND: Health policy-making can benefit from more effective use of research. In many policy settings there is scope to increase capacity for using research individually and organisationally, but little is known about what strategies work best in which circumstances. This review addresses the question: What causal mechanisms can best explain the observed outcomes of interventions that aim to increase policy-makers' capacity to use research in their work? METHODS: Articles were identified from three available reviews and two databases (PAIS and WoS; 1999-2016). Using a realist approach, articles were reviewed for information about contexts, outcomes (including process effects) and possible causal mechanisms. Strategy + Context + Mechanism = Outcomes (SCMO) configurations were developed, drawing on theory and findings from other studies to develop tentative hypotheses that might be applicable across a range of intervention sites. RESULTS: We found 22 studies that spanned 18 countries. There were two dominant design strategies (needs-based tailoring and multi-component design) and 18 intervention strategies targeting four domains of capacity, namely access to research, skills improvement, systems improvement and interaction. Many potential mechanisms were identified as well as some enduring contextual characteristics that all interventions should consider. The evidence was variable, but the SCMO analysis suggested that tailored interactive workshops supported by goal-focused mentoring, and genuine collaboration, seem particularly promising. Systems supports and platforms for cross-sector collaboration are likely to play crucial roles. Gaps in the literature are discussed. CONCLUSION: This exploratory review tentatively posits causal mechanisms that might explain how intervention strategies work in different contexts to build capacity for using research in policy-making.

Knowledge mobilisation for chronic disease prevention: the case of the Australian Prevention Partnership Centre.

BACKGROUND: Cross-sectoral, multidisciplinary partnership research is considered one of the most effective means of facilitating research-informed policy and practice, particularly for addressing complex problems such as chronic disease. Successful research partnerships tend to be underpinned by a range of features that enable knowledge mobilisation (KMb), seeking to connect academic researchers with decision-makers and practitioners to improve the nature, quality and use of research. This paper contributes to the growing discourse on partnership approaches by illustrating how knowledge mobilisation strategies are operationalised within the Australian Prevention Partnership Centre (the Centre), a national collaboration of academics, policy-makers and practitioners established to develop systems approaches for the prevention of lifestyle-related chronic diseases. METHODS: We undertook interviews with key academics, policy, and practice partners and funding representatives at the mid-point of the Centre's initial 5-year funding cycle. We aimed to explore how the Centre is functioning in practice, to develop a conceptual model of KMb within the Centre for use in further evaluation, and to identify ways of strengthening our approach to partnership research. Inductive and deductive thematic analysis was used to identify the key mechanisms underpinning the Centre's KMb approach. RESULTS: Six key mechanisms appeared to facilitate KMb within our Centre, namely Engagement, Partnerships, Co-production, Capacity and Skills, Knowledge Integration, and Adaptive Learning and Improvement. We developed a conceptual model that articulated these mechanisms in relation to the structures and processes that support them, as well as the Centre's goals. Findings also informed adaptations designed to strengthen the Centre. CONCLUSIONS: Findings provide insights into the practical realities of operationalising KMb strategies within a research partnership. Overall, the centre is perceived to be progressing towards its KMb goals, but challenges include stakeholders from different settings understanding each other's contexts and working together effectively, and ensuring knowledge generated across different projects within the Centre is integrated into a more comprehensive understanding of chronic disease prevention policy and practice. Our conceptual model is now informing ongoing developmental evaluation activities within the Centre, where it is being tested and refined.

Housing conditions of urban households with Aboriginal children in NSW Australia: tenure type matters.

BACKGROUND: Housing is a key determinant of the poor health of Aboriginal Australians. Most Aboriginal people live in cities and large towns, yet research into housing conditions has largely focused on those living in remote areas. This paper measures the prevalence of housing problems amongst participants in a study of urban Aboriginal families in New South Wales, Australia, and examines the relationship between tenure type and exposure to housing problems. METHODS: Cross-sectional survey data was provided by 600 caregivers of 1406 Aboriginal children aged 0-17 years participating in Phase One of the Study of Environment on Aboriginal Resilience and Child Health (SEARCH). Regression modelling of the associations between tenure type (own/mortgage, private rental or social housing) and housing problems was conducted, adjusting for sociodemographic factors. RESULTS: The majority (60%) of SEARCH households lived in social housing, 21% rented privately and 19% either owned their home outright or were paying a mortgage ("owned"). Housing problems were common, particularly structural problems, damp and mildew, vermin, crowding and unaffordability. Physical dwelling problems were most prevalent for those living in social housing, who were more likely to report three or more physical dwelling problems than those in owned (PR 3.19, 95%CI 1.97, 5.73) or privately rented homes (PR 1.49, 1.11, 2.08). However, those in social housing were the least likely to report affordability problems. Those in private rental moved home most frequently; children in private rental were more than three times as likely to have lived in four or more homes since birth than those in owned homes (PR 3.19, 95%CI 1.97, 5.73). Those in social housing were almost half as likely as those in private rental to have lived in four or more homes since birth (PR 0.56, 95%CI 0.14, 0.77). Crowding did not vary significantly by tenure type. CONCLUSIONS: The high prevalence of housing problems amongst study participants suggests that urban Aboriginal housing requires further attention as part of efforts to reduce the social and health disadvantage experienced by Aboriginal Australians. Particular attention should be directed to the needs of those renting in the private and social housing sectors, who are experiencing the poorest dwelling conditions.

Knowledge mobilisation for policy development: implementing systems approaches through participatory dynamic simulation modelling.

BACKGROUND: Evidence-based decision-making is an important foundation for health policy and service planning decisions, yet there remain challenges in ensuring that the many forms of available evidence are considered when decisions are being made. Mobilising knowledge for policy and practice is an emergent process, and one that is highly relational, often messy and profoundly context dependent. Systems approaches, such as dynamic simulation modelling can be used to examine both complex health issues and the context in which they are embedded, and to develop decision support tools. OBJECTIVE: This paper reports on the novel use of participatory simulation modelling as a knowledge mobilisation tool in Australian real-world policy settings. We describe how this approach combined systems science methodology and some of the core elements of knowledge mobilisation best practice. We describe the strategies adopted in three case studies to address both technical and socio-political issues, and compile the experiential lessons derived. Finally, we consider the implications of these knowledge mobilisation case studies and provide evidence for the feasibility of this approach in policy development settings. CONCLUSION: Participatory dynamic simulation modelling builds on contemporary knowledge mobilisation approaches for health stakeholders to collaborate and explore policy and health service scenarios for priority public health topics. The participatory methods place the decision-maker at the centre of the process and embed deliberative methods and co-production of knowledge. The simulation models function as health policy and programme dynamic decision support tools that integrate diverse forms of evidence, including research evidence, expert knowledge and localised contextual information. Further research is underway to determine the impact of these methods on health service decision-making.

Policymakers' experience of a capacity-building intervention designed to increase their use of research: a realist process evaluation.

BACKGROUND: An intervention's success depends on how participants interact with it in local settings. Process evaluation examines these interactions, indicating why an intervention was or was not effective, and how it (and similar interventions) can be improved for better contextual fit. This is particularly important for innovative trials like Supporting Policy In health with Research: an Intervention Trial (SPIRIT), where causal mechanisms are poorly understood. SPIRIT was testing a multi-component intervention designed to increase the capacity of health policymakers to use research. METHODS: Our mixed-methods process evaluation sought to explain variation in observed process effects across the six agencies that participated in SPIRIT. Data collection included observations of intervention workshops (n = 59), purposively sampled interviews (n = 76) and participant feedback forms (n = 553). Using a realist approach, data was coded for context-mechanism-process effect configurations (retroductive analysis) by two authors. RESULTS: Intervention workshops were very well received. There was greater variation of views regarding other aspects of SPIRIT such as data collection, communication and the intervention's overall value. We identified nine inter-related mechanisms that were crucial for engaging participants in these policy settings: (1) Accepting the premise (agreeing with the study's assumptions); (2) Self-determination (participative choice); (3) The Value Proposition (seeing potential gain); (4) 'Getting good stuff' (identifying useful ideas, resources or connections); (5) Self-efficacy (believing 'we can do this!'); (6) Respect (feeling that SPIRIT understands and values one's work); (7) Confidence (believing in the study's integrity and validity); (8) Persuasive leadership (authentic and compelling advocacy from leaders); and (9) Strategic insider facilitation (local translation and mediation). These findings were used to develop tentative explanatory propositions and to revise the programme theory. CONCLUSION: This paper describes how SPIRIT functioned in six policy agencies, including why strategies that worked well in one site were less effective in others. Findings indicate a complex interaction between participants' perception of the intervention, shifting contextual factors, and the form that the intervention took in each site. Our propositions provide transferable lessons about contextualised areas of strength and weakness that may be useful in the development and implementation of similar studies.

Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research.

BACKGROUND: Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers' capacity to engage with and use research. METHODS: We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales). RESULTS: Response rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal. CONCLUSIONS: Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.

Psychological distress in carers of Aboriginal children in urban New South Wales: findings from SEARCH (phase one).

OBJECTIVE: To examine the factors associated with psychological distress in parents and carers of Aboriginal children living in urban communities in New South Wales. DESIGN: Cross-sectional survey (phase one of the Study of Environment on Aboriginal Resilience and Child Health [SEARCH], November 2007 - December 2011). SETTING AND PARTICIPANTS: Primary care; 589 parents and carers of Aboriginal children were recruited when attending one of the four Aboriginal Community Controlled Health Services (ACCHSs) in urban NSW that participated in SEARCH. MAIN OUTCOME MEASURE: Kessler Psychological Distress Scale (K10) scores; a score of 22 or higher was deemed to indicate high levels of psychological distress. RESULTS: High levels of psychological distress were identified in 18% of our sample. The factors most strongly associated with this distress were functional limitations (v those with K10 scores under 22: adjusted odds ratio [aOR], 4.2; 95% CI, 1.3-13.5), previous hospitalisation (aOR, 5.5; 95% CI, 1.5-19.4) or other treatment for social and emotional wellbeing (aOR, 3.3; 95% CI, 1.3-8.4), low satisfaction with feeling part of the community (aOR, 0.83; 95% CI, 0.70-0.98) and low involvement in clubs and groups (aOR, 2.9; 95% CI, 1.2-7.3). CONCLUSIONS: Clinicians should note that those with functional limitations or a history of treatment for mental health problems are at higher risk of psychological distress and may require additional support. Increased funding that allows ACCHSs to provide mental health services, and funding and promoting programs and activities that increase social connectedness should remain focuses for ACCHSs and policy makers.

"There's a housing crisis going on in Sydney for Aboriginal people": focus group accounts of housing and perceived associations with health.

BACKGROUND: Poor housing is widely cited as an important determinant of the poor health status of Aboriginal Australians, as for indigenous peoples in other wealthy nations with histories of colonisation such as Canada, the United States of America and New Zealand. While the majority of Aboriginal Australians live in urban areas, most research into housing and its relationship with health has been conducted with those living in remote communities. This study explores the views of Aboriginal people living in Western Sydney about their housing circumstances and what relationships, if any, they perceive between housing and health. METHODS: Four focus groups were conducted with clients and staff of an Aboriginal community-controlled health service in Western Sydney (n = 38). Inductive, thematic analysis was conducted using framework data management methods in NVivo10. RESULTS: Five high-level themes were derived: the battle to access housing; secondary homelessness; overcrowding; poor dwelling conditions; and housing as a key determinant of health. Participants associated their challenging housing experiences with poor physical health and poor social and emotional wellbeing. Housing issues were said to affect people differently across the life course; participants expressed particular concern that poor housing was harming the health and developmental trajectories of many urban Aboriginal children. CONCLUSIONS: Housing was perceived as a pivotal determinant of health and wellbeing that either facilitates or hinders prospects for full and healthy lives. Many of the specific health concerns participants attributed to poor housing echo existing epidemiological research findings. These findings suggest that housing may be a key intervention point for improving the health of urban Aboriginal Australians.

The development of ORACLe: a measure of an organisation's capacity to engage in evidence-informed health policy.

BACKGROUND: Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations' research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations' capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership). METHOD: We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe. RESULTS: The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4). CONCLUSION: We utilised qualitative and quantitative methods to develop a system to assess and score organisations' capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.

Research impact: neither quick nor easy.

Greenhalgh and Fahy's paper about the 2014 Research Excellence Framework provides insights into the challenges of assessing research impact. Future research assessment exercises should consider how best to include measurement of indirect and non-linear impact and whether efforts in knowledge transfer and co-production should be explicitly recognised. Greenhalgh and Fahy's findings also demonstrate that the structure of the assessment exercise can privilege certain kinds of research and may therefore miss some research that has a high impact on policy and practice. There are a growing number of courses, tools, and funding models to assist researchers in making an impact, although as yet there is little evidence about whether these approaches work in practice. Please see related article: http://www.biomedcentral.com/1741-7015/13/232.

A narrative review of research impact assessment models and methods.

BACKGROUND: Research funding agencies continue to grapple with assessing research impact. Theoretical frameworks are useful tools for describing and understanding research impact. The purpose of this narrative literature review was to synthesize evidence that describes processes and conceptual models for assessing policy and practice impacts of public health research. METHODS: The review involved keyword searches of electronic databases, including MEDLINE, CINAHL, PsycINFO, EBM Reviews, and Google Scholar in July/August 2013. Review search terms included 'research impact', 'policy and practice', 'intervention research', 'translational research', 'health promotion', and 'public health'. The review included theoretical and opinion pieces, case studies, descriptive studies, frameworks and systematic reviews describing processes, and conceptual models for assessing research impact. The review was conducted in two phases: initially, abstracts were retrieved and assessed against the review criteria followed by the retrieval and assessment of full papers against review criteria. RESULTS: Thirty one primary studies and one systematic review met the review criteria, with 88% of studies published since 2006. Studies comprised assessments of the impacts of a wide range of health-related research, including basic and biomedical research, clinical trials, health service research, as well as public health research. Six studies had an explicit focus on assessing impacts of health promotion or public health research and one had a specific focus on intervention research impact assessment. A total of 16 different impact assessment models were identified, with the 'payback model' the most frequently used conceptual framework. Typically, impacts were assessed across multiple dimensions using mixed methodologies, including publication and citation analysis, interviews with principal investigators, peer assessment, case studies, and document analysis. The vast majority of studies relied on principal investigator interviews and/or peer review to assess impacts, instead of interviewing policymakers and end-users of research. CONCLUSIONS: Research impact assessment is a new field of scientific endeavour and there are a growing number of conceptual frameworks applied to assess the impacts of research.

Using conjoint analysis to develop a system of scoring policymakers' use of research in policy and program development.

BACKGROUND: The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers' research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts. METHODS: Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of research use. Generalised Estimating Equations were used to analyse respondents' choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts' ratings of extensive research use. RESULTS: Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation. CONCLUSIONS: We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

Using conjoint analysis to develop a system to score research engagement actions by health decision makers.

BACKGROUND: Effective use of research to inform policymaking can be strengthened by policymakers undertaking various research engagement actions (e.g., accessing, appraising, and applying research). Consequently, we developed a thorough measurement and scoring tool to assess whether and how policymakers undertook research engagement actions in the development of a policy document. This scoring tool breaks down each research engagement action into its key 'subactions' like a checklist. The primary aim was to develop the scoring tool further so that it assigned appropriate scores to each subaction based on its effectiveness for achieving evidence-informed policymaking. To establish the relative effectiveness of these subactions, we conducted a conjoint analysis, which was used to elicit the opinions and preferences of knowledge translation experts. METHOD: Fifty-four knowledge translation experts were recruited to undertake six choice surveys. Respondents were exposed to combinations of research engagement subactions called 'profiles', and rated on a 1-9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of each research engagement action. Generalised estimating equations were used to analyse respondents' choice data, where a utility coefficient was calculated for each subaction. A large utility coefficient indicates that a subaction was influential in guiding experts' ratings of extensive engagement with research. RESULTS: The calculated utilities were used as the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of engaging with research: searching academic literature databases, obtaining systematic reviews and peer-reviewed research, appraising relevance by verifying its applicability to the policy context, appraising quality by evaluating the validity of the method and conclusions, engaging in thorough collaborations with researchers, and undertaking formal research projects to inform the policy in question. CONCLUSIONS: We have generated an empirically-derived and context-sensitive method of measuring and scoring the extent to which policymakers engaged with research to inform policy development. The scoring system can be used by organisations to quantify staff research engagement actions and thus provide them with insights into what types of training, systems, and tools might improve their staff's research use capacity.

Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool.

BACKGROUND: There is a growing emphasis on the importance of research having demonstrable public benefit. Measurements of the impacts of research are therefore needed. We applied a modified impact assessment process that builds on best practice to 5 years (2003-2007) of intervention research funded by Australia's National Health and Medical Research Council to determine if these studies had post-research real-world policy and practice impacts. METHODS: We used a mixed method sequential methodology whereby chief investigators of eligible intervention studies who completed two surveys and an interview were included in our final sample (n = 50), on which we conducted post-research impact assessments. Data from the surveys and interviews were triangulated with additional information obtained from documentary analysis to develop comprehensive case studies. These case studies were then summarized and the reported impacts were scored by an expert panel using criteria for four impact dimensions: corroboration; attribution, reach, and importance. RESULTS: Nineteen (38%) of the cases in our final sample were found to have had policy and practice impacts, with an even distribution of high, medium, and low impact scores. While the tool facilitated a rigorous and explicit criterion-based assessment of post-research impacts, it was not always possible to obtain evidence using documentary analysis to corroborate the impacts reported in chief investigator interviews. CONCLUSIONS: While policy and practice is ideally informed by reviews of evidence, some intervention research can and does have real world impacts that can be attributed to single studies. We recommend impact assessments apply explicit criteria to consider the corroboration, attribution, reach, and importance of reported impacts on policy and practice. Impact assessments should also allow sufficient time between impact data collection and completion of the original research and include mechanisms to obtain end-user input to corroborate claims and reduce biases that result from seeking information from researchers only.

Increasing the scale and adoption of population health interventions: experiences and perspectives of policy makers, practitioners, and researchers.

BACKGROUND: Decisions to scale up population health interventions from small projects to wider state or national implementation is fundamental to maximising population-wide health improvements. The objectives of this study were to examine: i) how decisions to scale up interventions are currently made in practice; ii) the role that evidence plays in informing decisions to scale up interventions; and iii) the role policy makers, practitioners, and researchers play in this process. METHODS: Interviews with an expert panel of senior Australian and international public health policy-makers (n = 7), practitioners (n = 7), and researchers (n = 7) were conducted in May 2013 with a participation rate of 84%. RESULTS: Scaling up decisions were generally made through iterative processes and led by policy makers and/or practitioners, but ultimately approved by political leaders and/or senior executives of funding agencies. Research evidence formed a component of the overall set of information used in decision-making, but its contribution was limited by the paucity of relevant intervention effectiveness research, and data on costs and cost effectiveness. Policy makers, practitioners/service managers, and researchers had different, but complementary roles to play in the process of scaling up interventions. CONCLUSIONS: This analysis articulates the processes of how decisions to scale up interventions are made, the roles of evidence, and contribution of different professional groups. More intervention research that includes data on the effectiveness, reach, and costs of operating at scale and key service delivery issues (including acceptability and fit of interventions and delivery models) should be sought as this has the potential to substantially advance the relevance and ultimately usability of research evidence for scaling up population health action.

Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000-2006.

BACKGROUND: Intervention research provides important information regarding feasible and effective interventions for health policy makers, but few empirical studies have explored the mechanisms by which these studies influence policy and practice. This study provides an exploratory case series analysis of the policy, practice and other related impacts of the 15 research projects funded through the New South Wales Health Promotion Demonstration Research Grants Scheme during the period 2000 to 2006, and explored the factors mediating impacts. METHODS: Data collection included semi-structured interviews with the chief investigators (n = 17) and end-users (n = 29) of each of the 15 projects to explore if, how and under what circumstances the findings had been used, as well as bibliometric analysis and verification using documentary evidence. Data analysis involved thematic coding of interview data and triangulation with other data sources to produce case summaries of impacts for each project. Case summaries were then individually assessed against four impact criteria and discussed at a verification panel meeting where final group assessments of the impact of research projects were made and key influences of research impact identified. RESULTS: Funded projects had variable impacts on policy and practice. Project findings were used for agenda setting (raising awareness of issues), identifying areas and target groups for interventions, informing new policies, and supporting and justifying existing policies and programs across sectors. Reported factors influencing the use of findings were: i) nature of the intervention; ii) leadership and champions; iii) research quality; iv) effective partnerships; v) dissemination strategies used; and, vi) contextual factors. CONCLUSIONS: The case series analysis provides new insights into how and under what circumstances intervention research is used to influence real world policy and practice. The findings highlight that intervention research projects can achieve the greatest policy and practice impacts if they address proximal needs of the policy context by engaging end-users from the inception of projects and utilizing existing policy networks and structures, and using a range of strategies to disseminate findings that go beyond traditional peer review publications.

The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice.

Increased focus on prevention presents health promoters with new opportunities and challenges. In this context, the study of factors influencing policy-maker decisions to scale up health promotion interventions from small projects or controlled trials to wider state, national or international roll-out is increasingly important. This study aimed to: (i) examine the perspectives of senior researchers and policy-makers regarding concepts of 'scaling up' and 'scalability'; (ii) generate an agreed definition of 'scalability' and (iii) identify intervention and research design factors perceived to increase the potential for interventions to be implemented on a more widespread basis or 'scaled up'. A two-stage Delphi process with an expert panel of senior Australian public health intervention researchers (n = 7) and policy-makers (n = 7) and a review of relevant literature were conducted. Through this process 'scalability' was defined as: the ability of a health intervention shown to be efficacious on a small scale and or under controlled conditions to be expanded under real world conditions to reach a greater proportion of the eligible population, while retaining effectiveness. Results showed that in health promotion research insufficient attention is given to issues of effectiveness, reach and adoption; human, technical and organizational resources; costs; intervention delivery; contextual factors and appropriate evaluation approaches. If these issues were addressed in the funding, design and reporting of intervention research, it would advance the quality and usability of research for policy-makers and by doing so improve uptake and expansion of promising programs into practice.

Galvanizers, guides, champions, and shields: the many ways that policymakers use public health researchers.

CONTEXT: Public health researchers make a limited but important contribution to policy development. Some engage with policy directly through committees, advisory boards, advocacy coalitions, ministerial briefings, intervention design consultation, and research partnerships with government, as well as by championing research-informed policy in the media. Nevertheless, the research utilization literature has paid little attention to these diverse roles and the ways that policymakers use them. This article describes how policymakers use researchers in policymaking and examines how these activities relate to models of research utilization. It also explores the extent to which policymakers' accounts of using researchers concur with the experiences of "policy-engaged" public health researchers. METHODS: We conducted semi-structured interviews with thirty-two Australian civil servants, parliamentary ministers, and ministerial advisers identified as "research-engaged" by public health researchers. We used structured and inductive coding to generate categories that we then compared with some of the major research utilization models. FINDINGS: Policymakers were sophisticated and multifaceted users of researchers for purposes that we describe as Galvanizing Ideas, Clarification and Advice, Persuasion, and Defense. These categories overlapped but did not wholly fit with research utilization models. Despite the negative connotation, "being used" was reported as reciprocal and uncompromising, although researchers and policymakers were likely to categorize these uses differently. Policymakers countered views expressed by some researchers. That is, they sought robust dialogue and creative thinking rather than compliance, and they valued expert opinion when research was insufficient for decision making. The technical/political character of policy development shaped the ways in which researchers were used. CONCLUSIONS: Elucidating the diverse roles that public health researchers play in policymaking, and the multiple ways that policymakers use these roles, provides researchers and policymakers with a framework for negotiating and reflecting on activities that may advance the public health goals shared by both.