Impact of a health services innovation university program in a major public hospital and health service: a mixed methods evaluation.

BACKGROUND: While health services and their clinicians might seek to be innovative, finite budgets, increased demands on health services, and ineffective implementation strategies create challenges to sustaining innovation. These challenges can be addressed by building staff capacity to design cost-effective, evidence-based innovations, and selecting appropriate implementation strategies. A bespoke university award qualification and associated program of activities was developed to build the capacity of staff at Australia's largest health service to implement and evaluate evidence-based practice (EBP): a Graduate Certificate in Health Science majoring in Health Services Innovation. The aim of this study was to establish the health service's pre-program capacity to implement EBP and to identify preliminary changes in capacity that have occurred as a result of the Health Services Innovation program. METHODS: A mixed methods design underpinned by the Consolidated Framework for Implementation Research informed the research design, data collection, and analysis. Data about EBP implementation capacity aligned to the framework constructs were sought through qualitative interviews of university and health service executives, focus groups with students, and a quantitative survey of managers and students. The outcomes measured were knowledge of, attitudes towards, and use of EBP within the health service, as well as changes to practice which students identified had resulted from their participation in the program. RESULTS: The Health Services Innovation program has contributed to short-term changes in health service capacity to implement EBP. Participating students have not only increased their individual skills and knowledge, but also changed their EPB culture and practice which has ignited and sustained health service innovations and improvements in the first 18 months of the program. Capacity changes observed across wider sections of the organization include an increase in connections and networks, use of a shared language, and use of robust implementation science methods such as stakeholder analyses. CONCLUSION: This is a unique study that assessed data from all stakeholders: university and health service executives, students, and their managers. By assembling multiple perspectives, we identified that developing the social capital of the organization through delivering a full suite of capacity-building initiatives was critical to the preliminary success of the program.

A Systematic Review Protocol Investigating Community Gardening Impact Measures.

Existing community gardening research has tended to be exploratory and descriptive, utilising qualitative or mixed methodologies to explore and understand community garden participation. While research on community gardening attracts growing interest, the empirical rigour of measurement scales and embedded indicators has received comparatively less attention. Despite the extensive body of community gardening literature, a coherent narrative on valid, high quality approaches to the measurement of outcomes and impact across different cultural contexts is lacking and yet to be comprehensively examined. This is essential as cities are becoming hubs for cultural diversity. Systematic literature reviews that explore the multiple benefits of community gardening and other urban agriculture activities have been undertaken, however, a systematic review of the impact measures of community gardening is yet to be completed. This search protocol aims to address the following questions: (1) How are the health, wellbeing, social and environmental outcomes and impacts of community gardening measured? (2) What cultural diversity considerations have existing community garden measures taken into account? Demographic data will be collected along with clear domains/constructs of experiences, impacts and outcomes captured from previous literature to explore if evidence considers culturally heterogeneous and diverse populations. This will offer an understanding as to whether community gardening research is appropriately measuring this cross-cultural activity.

Applying an Implementation Framework to the Use of Evidence from Economic Evaluations in Making Healthcare Decisions.

BACKGROUND AND OBJECTIVE: There is a need for the application of theory in understanding the use of evidence from economic evaluations in healthcare decision making. The purpose of this study is to review the published literature on the use of evidence from economic evaluations for healthcare decision making and to map the findings to the Consolidated Framework for Implementation Research (CFIR). METHODS: A systematic search strategy was used to identify studies investigating the factors that determine the use of evidence from economic evaluation in healthcare decision making. Barriers and facilitators identified in the included studies were mapped across the five CFIR domains, with the "intervention" referring to the use of economic evaluations in decision making. Gaps, inconsistencies and emergent relations were identified through the mapping process. RESULTS: Fifty-three studies met eligibility criteria and were included in the review. The CFIR constructs associated with the Intervention Characteristics and those associated with the knowledge and beliefs of users of economic evaluations were widely cited in the identified barriers and facilitators. Other constructs from the CFIR had not been reported in the literature, such as 'organisational networks' and 'individual stage of change'. Most of the stages in the implementation process as described by the CFIR were reflected in the identified barriers and facilitators. DISCUSSION: By categorising barriers and facilitators into domains, the CFIR provides a systematic approach to assess how these factors interact. Literature gaps in the literature regarding the use of economic evaluation in healthcare decision making were identified, specifically issues regarding organisational networks and the role of feedback. CONCLUSIONS: Through mapping findings from studies of the use of evidence from economic evaluations in healthcare decision making, we present an implementation framework based on the CFIR for understanding the use of economic evaluations into practice.

Randomly auditing research labs could be an affordable way to improve research quality: A simulation study.

The "publish or perish" incentive drives many researchers to increase the quantity of their papers at the cost of quality. Lowering quality increases the number of false positive errors which is a key cause of the reproducibility crisis. We adapted a previously published simulation of the research world where labs that produce many papers are more likely to have "child" labs that inherit their characteristics. This selection creates a competitive spiral that favours quantity over quality. To try to halt the competitive spiral we added random audits that could detect and remove labs with a high proportion of false positives, and also improved the behaviour of "child" and "parent" labs who increased their effort and so lowered their probability of making a false positive error. Without auditing, only 0.2% of simulations did not experience the competitive spiral, defined by a convergence to the highest possible false positive probability. Auditing 1.35% of papers avoided the competitive spiral in 71% of simulations, and auditing 1.94% of papers in 95% of simulations. Audits worked best when they were only applied to established labs with 50 or more papers compared with labs with 25 or more papers. Adding a ±20% random error to the number of false positives to simulate peer reviewer error did not reduce the audits' efficacy. The main benefit of the audits was via the increase in effort in "child" and "parent" labs. Audits improved the literature by reducing the number of false positives from 30.2 per 100 papers to 12.3 per 100 papers. Auditing 1.94% of papers would cost an estimated $15.9 million per year if applied to papers produced by National Institutes of Health funding. Our simulation greatly simplifies the research world and there are many unanswered questions about if and how audits would work that can only be addressed by a trial of an audit.

Does engagement predict research use? An analysis of The Conversation Annual Survey 2016.

The impact of research on the world beyond academia has increasingly become an area of focus in research performance assessments internationally. Impact assessment is expected to incentivise researchers to increase engagement with industry, government and the public more broadly. Increased engagement is in turn expected to increase translation of research so decision-makers can use research to inform development of policies, programs, practices, processes, products, and other mechanisms, through which impact can be realised. However, research has shown that various factors affect research use, and evidence on 'what works' to increase decision-makers' use of research is limited. The Conversation is an open access research communication platform, published under Creative Commons licence, which translates research into news articles to engage a general audience, aiming to improve understanding of current issues and complex social problems. To identify factors that predict use of academic research and expertise reported in The Conversation, regression analyses were performed using The Conversation Australia 2016 Annual Survey data. A broad range of factors predicted use, with engagement actions being the most common. Interestingly, different types of engagement actions predicted different types of use. This suggests that to achieve impact through increased engagement, a deeper understanding of how and why different engagement actions elicit different types of use is needed. Findings also indicate The Conversation is overcoming some of the most commonly identified barriers to the use of research: access, relevance, actionable outcomes, and timeliness. As such, The Conversation offers an effective model for providing access to and communicating research in a way that enables use, a necessary precursor to achieving research impact.

Type, frequency and purpose of information used to inform public health policy and program decision-making.

BACKGROUND: There is a growing demand for researchers to document the impact of research to demonstrate how it contributes to community outcomes. In the area of public health it is expected that increases in the use of research to inform policy and program development will lead to improved public health outcomes. To determine whether research has an impact on public health outcomes, we first need to assess to what extent research has been used and how it has been used. However, there are relatively few studies to date that have quantitatively measured the extent and purpose of use of research in public health policy environments. This study sought to quantitatively measure the frequency and purpose of use of research evidence in comparison to use of other information types in a specific public health policy environment, workplace and transport injury prevention and rehabilitation compensation. METHODS: A survey was developed to measure the type, frequency and purpose of information used to inform policy and program decision-making. RESULTS: Research evidence was the type of information used least frequently and internal data and reports was the information type used most frequently. Findings also revealed differences in use of research between and within the two government public health agencies studied. In particular the main focus of participants' day-to-day role was associated with the type of information used. Research was used mostly for conceptual purposes. Interestingly, research was used for instrumental purposes more often than it was used for symbolic purposes, which is contrary to findings of previous research. CONCLUSIONS: These results have implications for the design and implementation of research translation interventions in the context within which the study was undertaken. In particular, they suggest that intervention will need to be targeted to the information needs of the different role groups within an organisation. The results can also be utilised as a baseline measure for intervention evaluations and assessments of research impact in this context.

Predicting research use in a public health policy environment: results of a logistic regression analysis.

BACKGROUND: Use of research evidence in public health policy decision-making is affected by a range of contextual factors operating at the individual, organisational and external levels. Context-specific research is needed to target and tailor research translation intervention design and implementation to ensure that factors affecting research in a specific context are addressed. Whilst such research is increasing, there remain relatively few studies that have quantitatively assessed the factors that predict research use in specific public health policy environments. METHOD: A quantitative survey was designed and implemented within two public health policy agencies in the Australian state of Victoria. Binary logistic regression analyses were conducted on survey data provided by 372 participants. Univariate logistic regression analyses of 49 factors revealed 26 factors that significantly predicted research use independently. The 26 factors were then tested in a single model and five factors emerged as significant predictors of research over and above all other factors. RESULTS: The five key factors that significantly predicted research use were the following: relevance of research to day-to-day decision-making, skills for research use, internal prompts for use of research, intention to use research within the next 12 months and the agency for which the individual worked. CONCLUSIONS: These findings suggest that individual- and organisational-level factors are the critical factors to target in the design of interventions aiming to increase research use in this context. In particular, relevance of research and skills for research use would be necessary to target. The likelihood for research use increased 11- and 4-fold for those who rated highly on these factors. This study builds on previous research and contributes to the currently limited number of quantitative studies that examine use of research evidence in a large sample of public health policy and program decision-makers within a specific context. The survey used in this study is likely to be relevant for use in other public health policy contexts.

Measuring use of research evidence in public health policy: a policy content analysis.

BACKGROUND: There are few Australian studies showing how research evidence is used to inform the development of public health policy. International research has shown that compensation for injury rehabilitation can have negative impacts on health outcomes. This study examined transport injury compensation policy in the Australian state of Victoria to: determine type and purpose of reference to information sources; and to identify the extent of reference to academic research evidence in transport related injury rehabilitation compensation policy. METHODS: Quantitative content analysis of injury rehabilitation compensation policies (N = 128) from the Victorian state government transport accident compensation authority. RESULTS: The most commonly referenced types of information were Internal Policy (median = 6 references per policy), Clinical/Medical (2.5), and Internal Legislation (1). Academic Research Evidence was the least often referenced source of information. The main purpose of reference to information was to support injury treatment and rehabilitation compensation claims decision-making. CONCLUSIONS: Transport injury compensation policy development is complex; with multiple sources of information cited including legislation, internal policy, external policy and clinical/medical evidence. There is limited use of academic research evidence in Victorian state government injury treatment and rehabilitation compensation policies. Decisions regarding compensation for injury treatment and rehabilitation services could benefit from greater use of academic research evidence. This study is one of the first to examine the use of research evidence in existing Australian public health policy decision-making using rigorous quantitative methods. It provides a practical example of how use of research evidence in public health policy can be objectively measured.

External factors affecting decision-making and use of evidence in an Australian public health policy environment.

This study examined external factors affecting policy and program decision-making in a specific public health policy context: injury prevention and rehabilitation compensation in the Australian state of Victoria. The aim was twofold: identify external factors that affect policy and program decision-making in this specific context; use this evidence to inform targeting of interventions aimed at increasing research use in this context. Qualitative interviews were undertaken from June 2011 to January 2012 with 33 employees from two state government agencies. Key factors identified were stakeholder feedback and action, government and ministerial input, legal feedback and action, injured persons and the media. The identified external factors were able to significantly influence policy and program decision-making processes: acting as both barriers and facilitators, depending on the particular issue at hand. The factors with the most influence were the Minister and government, lawyers, and agency stakeholders, particularly health providers, trade unions and employer groups. This research revealed that interventions aimed at increasing use of research in this context must target and harness the influence of these groups. This research provides critical insights for researchers seeking to design interventions to increase use of research in policy environments and influence decision-making in Victorian injury prevention and rehabilitation compensation.