Organisational capacity and its relationship to research use in six Australian health policy agencies.

There are calls for policymakers to make greater use of research when formulating policies. Therefore, it is important that policy organisations have a range of tools and systems to support their staff in using research in their work. The aim of the present study was to measure the extent to which a range of tools and systems to support research use were available within six Australian agencies with a role in health policy, and examine whether this was related to the extent of engagement with, and use of research in policymaking by their staff. The presence of relevant systems and tools was assessed via a structured interview called ORACLe which is conducted with a senior executive from the agency. To measure research use, four policymakers from each agency undertook a structured interview called SAGE, which assesses and scores the extent to which policymakers engaged with (i.e., searched for, appraised, and generated) research, and used research in the development of a specific policy document. The results showed that all agencies had at least a moderate range of tools and systems in place, in particular policy development processes; resources to access and use research (such as journals, databases, libraries, and access to research experts); processes to generate new research; and mechanisms to establish relationships with researchers. Agencies were less likely, however, to provide research training for staff and leaders, or to have evidence-based processes for evaluating existing policies. For the majority of agencies, the availability of tools and systems was related to the extent to which policymakers engaged with, and used research when developing policy documents. However, some agencies did not display this relationship, suggesting that other factors, namely the organisation's culture towards research use, must also be considered.

Do policy-makers find commissioned rapid reviews useful?

BACKGROUND: Rapid reviews are increasingly used by policy agencies to access relevant research in short timeframes. Despite the growing number of programmes, little is known about how rapid reviews are used by health policy agencies. This study examined whether and how rapid reviews commissioned using a knowledge brokering programme were used by Australian policy-makers. METHODS: This study used interview data to examine the use of 139 rapid reviews by health policy agencies that were commissioned between 2006 and 2015. Transcripts were coded to identify how rapid reviews were used, the type of policy processes in which they were used, what evidence of use was provided and what reasons were given when rapid reviews were not used. Fisher's exact test was used to assess variation between types of agencies. RESULTS: Overall, 89% of commissioned rapid reviews were used by the commissioning agencies and 338 separate instances of use were identified, namely, on average, three uses per review. Policy-makers used reviews primarily to determine the details of a policy or programme, identify priorities for future action or investment, negotiate interjurisdictional decisions, evaluate alternative solutions for a policy problem, and communicate information to stakeholders. Some variation in use was observed across agencies. Reasons for non-use were related to changes in organisational structures, resources or key personnel in the commissioning agencies, or changes in the broader political environment. CONCLUSIONS: This study found that almost all rapid reviews had been used by the agencies who commissioned them, primarily in policy and programme development, agenda-setting, and to communicate information to stakeholders. Reviews were used mostly in instrumental and conceptual ways and there was little evidence of symbolic use. Variations in use were identified across agencies. The findings suggest that commissioned rapid reviews are an effective means of providing timely relevant research for use in policy processes and that review findings may be applied in a variety of ways.

Improving the quality of healthcare: a cross-sectional study of the features of successful clinical networks.

OBJECTIVES: Networks of clinical experts are being established internationally to help embed evidence based care in health systems. There is emerging evidence that these clinical networks can drive quality improvement programs, but the features that distinguish successful networks are largely unknown. We examined the factors that make clinical networks effective at improving quality of care and facilitating system-wide changes. METHODS: We conducted a retrospective cross-sectional study of 19 state-wide clinical networks that reflected a range of medical and surgical specialty care and were in operation from 2006 to 2008 in New South Wales, Australia. We conducted qualitative interviews with network leaders to characterise potential impacts, and conducted internet surveys of network members to evaluate external support and the organisational and program characteristics of their respective networks. The main outcome measures were median ratings of individual network impacts on quality of care and system-wide changes, determined through independent assessment of documented evidence by an expert panel. RESULTS: We interviewed 19 network managers and 32 network co-chairs; 592 network members completed internet surveys. Three networks were rated as having had high impact on quality of care, and seven as having had high impact on system-wide change. Better-perceived strategic and operational network management was significantly associated with higher ratings of impact on quality of care (coefficient estimate 0.86; 95% confidence interval [CI] 0.02, 1.69). Better-perceived leadership of the network manager (coefficient estimate 0.47; 95% CI 0.10, 0.85) and strategic and operational network management (coefficient estimate 0.23; 95% CI 0.06, 0.41) were associated with higher ratings of impact on system-wide change. CONCLUSIONS: This study represents the largest study of clinical networks undertaken to date. The results suggest that clinical networks that span the health system can improve quality of care and facilitate system-wide change. Network management and leadership, encompassing both strategic and operational elements at the organisational level, appear to be the primary influences on network success. These findings can guide future organisational and system-wide change programs and the development or strengthening of clinical networks to help implement evidence based care to improve service delivery and outcomes.

The EXpert PANel Decision (EXPAND) method: a way to measure the impact of diverse quality improvement activities of clinical networks.

Objectives and importance of study: Evaluating impacts of quality improvement activities across diverse clinical focus areas is challenging. However, evaluation is necessary to determine if the activities had an impact on quality of care and resulted in system-wide change. Clinical networks of health providers aim to provide a platform for accelerating quality improvement activities and adopting evidence based practices. However, most networks do not collect primary data that would enable evaluation of impact. We adapted an established expert panel approach to measure the impacts of efforts in 19 clinical networks to improve care and promote health system change, to determine whether these efforts achieved their purpose. STUDY TYPE: A retrospective cross-sectional study of 19 clinical networks using multiple methods of data collection including the EXpert PANel Decision (EXPAND) method. METHODS: Network impacts were identified through interviews with network managers (n = 19) and co-chairs (n = 32), and document review. The EXPAND method brought together five independent experts who provided initial individual ratings of overall network impact. After attendance at an in-person moderated meeting where aggregate scores were discussed, the experts provided a final rating. Median scores of postmeeting ratings were the final measures of network impact. RESULTS: Among the 19 clinical networks, experts rated 47% (n = 9) as having a limited impact on improving quality of care, 37% (n = 7) as having a moderate impact and 16% (n = 3) as having a high impact. The experts rated 26% (n = 5) of clinical networks as having a limited impact on facilitating system-wide change, 37% (n = 7) as having a moderate impact and 37% (n = 7) as having a high impact. CONCLUSION: The EXPAND method enabled appraisal of diverse clinical networks in the absence of primary data that could directly evaluate network impacts. The EXPAND method can be applied to assess the impact of quality improvement initiatives across diverse clinical areas to inform healthcare planning, delivery and performance. Further research is needed to assess its reliability and validity.

Policymakers' experience of a capacity-building intervention designed to increase their use of research: a realist process evaluation.

BACKGROUND: An intervention's success depends on how participants interact with it in local settings. Process evaluation examines these interactions, indicating why an intervention was or was not effective, and how it (and similar interventions) can be improved for better contextual fit. This is particularly important for innovative trials like Supporting Policy In health with Research: an Intervention Trial (SPIRIT), where causal mechanisms are poorly understood. SPIRIT was testing a multi-component intervention designed to increase the capacity of health policymakers to use research. METHODS: Our mixed-methods process evaluation sought to explain variation in observed process effects across the six agencies that participated in SPIRIT. Data collection included observations of intervention workshops (n = 59), purposively sampled interviews (n = 76) and participant feedback forms (n = 553). Using a realist approach, data was coded for context-mechanism-process effect configurations (retroductive analysis) by two authors. RESULTS: Intervention workshops were very well received. There was greater variation of views regarding other aspects of SPIRIT such as data collection, communication and the intervention's overall value. We identified nine inter-related mechanisms that were crucial for engaging participants in these policy settings: (1) Accepting the premise (agreeing with the study's assumptions); (2) Self-determination (participative choice); (3) The Value Proposition (seeing potential gain); (4) 'Getting good stuff' (identifying useful ideas, resources or connections); (5) Self-efficacy (believing 'we can do this!'); (6) Respect (feeling that SPIRIT understands and values one's work); (7) Confidence (believing in the study's integrity and validity); (8) Persuasive leadership (authentic and compelling advocacy from leaders); and (9) Strategic insider facilitation (local translation and mediation). These findings were used to develop tentative explanatory propositions and to revise the programme theory. CONCLUSION: This paper describes how SPIRIT functioned in six policy agencies, including why strategies that worked well in one site were less effective in others. Findings indicate a complex interaction between participants' perception of the intervention, shifting contextual factors, and the form that the intervention took in each site. Our propositions provide transferable lessons about contextualised areas of strength and weakness that may be useful in the development and implementation of similar studies.

Housing conditions of urban households with Aboriginal children in NSW Australia: tenure type matters.

BACKGROUND: Housing is a key determinant of the poor health of Aboriginal Australians. Most Aboriginal people live in cities and large towns, yet research into housing conditions has largely focused on those living in remote areas. This paper measures the prevalence of housing problems amongst participants in a study of urban Aboriginal families in New South Wales, Australia, and examines the relationship between tenure type and exposure to housing problems. METHODS: Cross-sectional survey data was provided by 600 caregivers of 1406 Aboriginal children aged 0-17 years participating in Phase One of the Study of Environment on Aboriginal Resilience and Child Health (SEARCH). Regression modelling of the associations between tenure type (own/mortgage, private rental or social housing) and housing problems was conducted, adjusting for sociodemographic factors. RESULTS: The majority (60%) of SEARCH households lived in social housing, 21% rented privately and 19% either owned their home outright or were paying a mortgage ("owned"). Housing problems were common, particularly structural problems, damp and mildew, vermin, crowding and unaffordability. Physical dwelling problems were most prevalent for those living in social housing, who were more likely to report three or more physical dwelling problems than those in owned (PR 3.19, 95%CI 1.97, 5.73) or privately rented homes (PR 1.49, 1.11, 2.08). However, those in social housing were the least likely to report affordability problems. Those in private rental moved home most frequently; children in private rental were more than three times as likely to have lived in four or more homes since birth than those in owned homes (PR 3.19, 95%CI 1.97, 5.73). Those in social housing were almost half as likely as those in private rental to have lived in four or more homes since birth (PR 0.56, 95%CI 0.14, 0.77). Crowding did not vary significantly by tenure type. CONCLUSIONS: The high prevalence of housing problems amongst study participants suggests that urban Aboriginal housing requires further attention as part of efforts to reduce the social and health disadvantage experienced by Aboriginal Australians. Particular attention should be directed to the needs of those renting in the private and social housing sectors, who are experiencing the poorest dwelling conditions.

Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research.

BACKGROUND: Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers' capacity to engage with and use research. METHODS: We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales). RESULTS: Response rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal. CONCLUSIONS: Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.

Does knowledge brokering improve the quality of rapid review proposals? A before and after study.

BACKGROUND: Rapid reviews are increasingly being used to help policy makers access research in short time frames. A clear articulation of the review's purpose, questions, scope, methods and reporting format is thought to improve the quality and generalisability of review findings. The aim of the study is to explore the effectiveness of knowledge brokering in improving the perceived clarity of rapid review proposals from the perspective of potential reviewers. To conduct the study, we drew on the Evidence Check program, where policy makers draft a review proposal (a pre knowledge brokering proposal) and have a 1-hour session with a knowledge broker, who re-drafts the proposal based on the discussion (a post knowledge brokering proposal). METHODS: We asked 30 reviewers who had previously undertaken Evidence Check reviews to examine the quality of 60 pre and 60 post knowledge brokering proposals. Reviewers were blind to whether the review proposals they received were pre or post knowledge brokering. Using a six-point Likert scale, reviewers scored six questions examining clarity of information about the review's purpose, questions, scope, method and format and reviewers' confidence that they could meet policy makers' needs. Each reviewer was allocated two pre and two post knowledge brokering proposals, randomly ordered, from the 60 reviews, ensuring no reviewer received a pre and post knowledge brokering proposal from the same review. RESULTS: The results showed that knowledge brokering significantly improved the scores for all six questions addressing the perceived clarity of the review proposal and confidence in meeting policy makers' needs; with average changes of 0.68 to 1.23 from pre to post across the six domains. CONCLUSIONS: This study found that knowledge brokering increased the perceived clarity of information provided in Evidence Check rapid review proposals and the confidence of reviewers that they could meet policy makers' needs. Further research is needed to identify how the knowledge brokering process achieves these improvements and to test the applicability of the findings in other rapid review programs.

A case study of enhanced clinical care enabled by Aboriginal health research: the Hearing, EAr health and Language Services (HEALS) project.

OBJECTIVE: To describe and evaluate Hearing EAr health and Language Services (HEALS), a New South Wales (NSW) health initiative implemented in 2013 and 2014 as a model for enhanced clinical services arising from Aboriginal health research. METHODS: A case-study involving a mixed-methods evaluation of the origins and outcomes of HEALS, a collaboration among five NSW Aboriginal Community Controlled Health Services (ACCHS), the Sydney Children's Hospitals Network, NSW Health, the Aboriginal Health and Medical Research Council, and local service providers. Service delivery data was collected fortnightly; semi-structured interviews were conducted with healthcare providers and caregivers of children who participated in HEALS. RESULTS: To circumvent health service barriers, HEALS used relationships established through the Study of Environment on Aboriginal Resilience and Child Health (SEARCH) to form a specialist healthcare network. HEALS employed dedicated staff and provided a Memorandum of Understanding (detailing mutual goals and responsibilities) for each ACCHS. Despite very tight timeframes, HEALS provided services for 653 Aboriginal children, including 5,822 speech-language pathology sessions and 219 Ear, Nose and Throat procedures. Four themes reflecting the perceived impact of HEALS were identified: valued clinical outcomes, raising community awareness, developing relationships/networks and augmented service delivery. CONCLUSIONS: HEALS delivered rapid and effective specialist healthcare services through an existing research collaboration with five ACCHS, cooperation from local health service providers, and effective community engagement. Implications for Public Health: HEALS serves as a framework for targeted, enhanced healthcare that benefits Aboriginal communities by encapsulating the 'no research without service' philosophy.

The Study of Environment on Aboriginal Resilience and Child Health (SEARCH): a long-term platform for closing the gap.

The full potential for research to improve Aboriginal health has not yet been realised. This paper describes an established long-term action partnership between Aboriginal Community Controlled Health Services (ACCHSs), the Aboriginal Health and Medical Research Council of New South Wales (AH&MRC), researchers and the Sax Institute, which is committed to using high-quality data to bring about health improvements through better services, policies and programs. The ACCHSs, in particular, have ensured that the driving purpose of the research conducted is to stimulate action to improve health for urban Aboriginal children and their families. This partnership established a cohort study of 1600 urban Aboriginal children and their caregivers, known as SEARCH (the Study of Environment on Aboriginal Resilience and Child Health), which is now having significant impacts on health, services and programs for urban Aboriginal children and their families. This paper describes some examples of the impacts of SEARCH, and reflects on the ways of working that have enabled these changes to occur, such as strong governance, a focus on improved health, AH&MRC and ACCHS leadership, and strategies to support the ACCHS use of data and to build Aboriginal capacity.

"There's a housing crisis going on in Sydney for Aboriginal people": focus group accounts of housing and perceived associations with health.

BACKGROUND: Poor housing is widely cited as an important determinant of the poor health status of Aboriginal Australians, as for indigenous peoples in other wealthy nations with histories of colonisation such as Canada, the United States of America and New Zealand. While the majority of Aboriginal Australians live in urban areas, most research into housing and its relationship with health has been conducted with those living in remote communities. This study explores the views of Aboriginal people living in Western Sydney about their housing circumstances and what relationships, if any, they perceive between housing and health. METHODS: Four focus groups were conducted with clients and staff of an Aboriginal community-controlled health service in Western Sydney (n = 38). Inductive, thematic analysis was conducted using framework data management methods in NVivo10. RESULTS: Five high-level themes were derived: the battle to access housing; secondary homelessness; overcrowding; poor dwelling conditions; and housing as a key determinant of health. Participants associated their challenging housing experiences with poor physical health and poor social and emotional wellbeing. Housing issues were said to affect people differently across the life course; participants expressed particular concern that poor housing was harming the health and developmental trajectories of many urban Aboriginal children. CONCLUSIONS: Housing was perceived as a pivotal determinant of health and wellbeing that either facilitates or hinders prospects for full and healthy lives. Many of the specific health concerns participants attributed to poor housing echo existing epidemiological research findings. These findings suggest that housing may be a key intervention point for improving the health of urban Aboriginal Australians.

The development of ORACLe: a measure of an organisation's capacity to engage in evidence-informed health policy.

BACKGROUND: Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations' research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations' capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership). METHOD: We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe. RESULTS: The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4). CONCLUSION: We utilised qualitative and quantitative methods to develop a system to assess and score organisations' capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.

A mixed methods study of the factors that influence whether intervention research has policy and practice impacts: perceptions of Australian researchers.

OBJECTIVES: To investigate researchers' perceptions about the factors that influenced the policy and practice impacts (or lack of impact) of one of their own funded intervention research studies. DESIGN: Mixed method, cross-sectional study. SETTING: Intervention research conducted in Australia and funded by Australia's National Health and Medical Research Council between 2003 and 2007. PARTICIPANTS: The chief investigators from 50 funded intervention research studies were interviewed to determine if their study had achieved policy and practice impacts, how and why these impacts had (or had not) occurred and the approach to dissemination they had employed. RESULTS: We found that statistically significant intervention effects and publication of results influenced whether there were policy and practice impacts, along with factors related to the nature of the intervention itself, the researchers' experience and connections, their dissemination and translation efforts, and the postresearch context. CONCLUSIONS: This study indicates that sophisticated approaches to intervention development, dissemination actions and translational efforts are actually widespread among experienced researches, and can achieve policy and practice impacts. However, it was the links between the intervention results, further dissemination actions by researchers and a variety of postresearch contextual factors that ultimately determined whether a study had policy and practice impacts. Given the complicated interplay between the various factors, there appears to be no simple formula for determining which intervention studies should be funded in order to achieve optimal policy and practice impacts.

The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy.

The recent proliferation of strategies designed to increase the use of research in health policy (knowledge exchange) demands better application of contemporary conceptual understandings of how research shapes policy. Predictive models, or action frameworks, are needed to organise existing knowledge and enable a more systematic approach to the selection and testing of intervention strategies. Useful action frameworks need to meet four criteria: have a clearly articulated purpose; be informed by existing knowledge; provide an organising structure to build new knowledge; and be capable of guiding the development and testing of interventions. This paper describes the development of the SPIRIT Action Framework. A literature search and interviews with policy makers identified modifiable factors likely to influence the use of research in policy. An iterative process was used to combine these factors into a pragmatic tool which meets the four criteria. The SPIRIT Action Framework can guide conceptually-informed practical decisions in the selection and testing of interventions to increase the use of research in policy. The SPIRIT Action Framework hypothesises that a catalyst is required for the use of research, the response to which is determined by the capacity of the organisation to engage with research. Where there is sufficient capacity, a series of research engagement actions might occur that facilitate research use. These hypotheses are being tested in ongoing empirical work.

A narrative review of research impact assessment models and methods.

BACKGROUND: Research funding agencies continue to grapple with assessing research impact. Theoretical frameworks are useful tools for describing and understanding research impact. The purpose of this narrative literature review was to synthesize evidence that describes processes and conceptual models for assessing policy and practice impacts of public health research. METHODS: The review involved keyword searches of electronic databases, including MEDLINE, CINAHL, PsycINFO, EBM Reviews, and Google Scholar in July/August 2013. Review search terms included 'research impact', 'policy and practice', 'intervention research', 'translational research', 'health promotion', and 'public health'. The review included theoretical and opinion pieces, case studies, descriptive studies, frameworks and systematic reviews describing processes, and conceptual models for assessing research impact. The review was conducted in two phases: initially, abstracts were retrieved and assessed against the review criteria followed by the retrieval and assessment of full papers against review criteria. RESULTS: Thirty one primary studies and one systematic review met the review criteria, with 88% of studies published since 2006. Studies comprised assessments of the impacts of a wide range of health-related research, including basic and biomedical research, clinical trials, health service research, as well as public health research. Six studies had an explicit focus on assessing impacts of health promotion or public health research and one had a specific focus on intervention research impact assessment. A total of 16 different impact assessment models were identified, with the 'payback model' the most frequently used conceptual framework. Typically, impacts were assessed across multiple dimensions using mixed methodologies, including publication and citation analysis, interviews with principal investigators, peer assessment, case studies, and document analysis. The vast majority of studies relied on principal investigator interviews and/or peer review to assess impacts, instead of interviewing policymakers and end-users of research. CONCLUSIONS: Research impact assessment is a new field of scientific endeavour and there are a growing number of conceptual frameworks applied to assess the impacts of research.

Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool.

BACKGROUND: There is a growing emphasis on the importance of research having demonstrable public benefit. Measurements of the impacts of research are therefore needed. We applied a modified impact assessment process that builds on best practice to 5 years (2003-2007) of intervention research funded by Australia's National Health and Medical Research Council to determine if these studies had post-research real-world policy and practice impacts. METHODS: We used a mixed method sequential methodology whereby chief investigators of eligible intervention studies who completed two surveys and an interview were included in our final sample (n = 50), on which we conducted post-research impact assessments. Data from the surveys and interviews were triangulated with additional information obtained from documentary analysis to develop comprehensive case studies. These case studies were then summarized and the reported impacts were scored by an expert panel using criteria for four impact dimensions: corroboration; attribution, reach, and importance. RESULTS: Nineteen (38%) of the cases in our final sample were found to have had policy and practice impacts, with an even distribution of high, medium, and low impact scores. While the tool facilitated a rigorous and explicit criterion-based assessment of post-research impacts, it was not always possible to obtain evidence using documentary analysis to corroborate the impacts reported in chief investigator interviews. CONCLUSIONS: While policy and practice is ideally informed by reviews of evidence, some intervention research can and does have real world impacts that can be attributed to single studies. We recommend impact assessments apply explicit criteria to consider the corroboration, attribution, reach, and importance of reported impacts on policy and practice. Impact assessments should also allow sufficient time between impact data collection and completion of the original research and include mechanisms to obtain end-user input to corroborate claims and reduce biases that result from seeking information from researchers only.

The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice.

Increased focus on prevention presents health promoters with new opportunities and challenges. In this context, the study of factors influencing policy-maker decisions to scale up health promotion interventions from small projects or controlled trials to wider state, national or international roll-out is increasingly important. This study aimed to: (i) examine the perspectives of senior researchers and policy-makers regarding concepts of 'scaling up' and 'scalability'; (ii) generate an agreed definition of 'scalability' and (iii) identify intervention and research design factors perceived to increase the potential for interventions to be implemented on a more widespread basis or 'scaled up'. A two-stage Delphi process with an expert panel of senior Australian public health intervention researchers (n = 7) and policy-makers (n = 7) and a review of relevant literature were conducted. Through this process 'scalability' was defined as: the ability of a health intervention shown to be efficacious on a small scale and or under controlled conditions to be expanded under real world conditions to reach a greater proportion of the eligible population, while retaining effectiveness. Results showed that in health promotion research insufficient attention is given to issues of effectiveness, reach and adoption; human, technical and organizational resources; costs; intervention delivery; contextual factors and appropriate evaluation approaches. If these issues were addressed in the funding, design and reporting of intervention research, it would advance the quality and usability of research for policy-makers and by doing so improve uptake and expansion of promising programs into practice.

Determinants of successful clinical networks: the conceptual framework and study protocol.

BACKGROUND: Clinical networks are increasingly being viewed as an important strategy for increasing evidence-based practice and improving models of care, but success is variable and characteristics of networks with high impact are uncertain. This study takes advantage of the variability in the functioning and outcomes of networks supported by the Australian New South Wales (NSW) Agency for Clinical Innovation's non-mandatory model of clinical networks to investigate the factors that contribute to the success of clinical networks. METHODS/DESIGN: The objective of this retrospective study is to examine the association between external support, organisational and program factors, and indicators of success among 19 clinical networks over a three-year period (2006-2008). The outcomes (health impact, system impact, programs implemented, engagement, user perception, and financial leverage) and explanatory factors will be collected using a web-based survey, interviews, and record review. An independent expert panel will provide judgements about the impact or extent of each network's initiatives on health and system impacts. The ratings of the expert panel will be the outcome used in multivariable analyses. Following the rating of network success, a qualitative study will be conducted to provide a more in-depth examination of the most successful networks. DISCUSSION: This is the first study to combine quantitative and qualitative methods to examine the factors that contribute to the success of clinical networks and, more generally, is the largest study of clinical networks undertaken. The adaptation of expert panel methods to rate the impacts of networks is the methodological innovation of this study. The proposed project will identify the conditions that should be established or encouraged by agencies developing clinical networks and will be of immediate use in forming strategies and programs to maximise the effectiveness of such networks.