Evaluating Research for Development: Innovation to Navigate Complexity.

Large publicly funded programmes of research continue to receive increased investment as interventions aiming to produce impact for the world's poorest and most marginalized populations. At this intersection of research and development, research is expected to contribute to complex processes of societal change. Embracing a co-produced view of impact as emerging along uncertain causal pathways often without predefined outcomes calls for innovation in the use of complexity-aware approaches to evaluation. The papers in this special issue present rich experiences of authors working across sectors and geographies, employing methodological innovation and navigating power as they reconcile tensions. They illustrate the challenges with (i) evaluating performance to meet accountability demands while fostering learning for adaptation; (ii) evaluating prospective theories of change while capturing emergent change; (iii) evaluating internal relational dimensions while measuring external development outcomes; (iv) evaluating across scales: from measuring local level end impact to understanding contributions to systems level change. Taken as a whole, the issue illustrates how the research for development evaluation field is maturing through the experiences of a growing and diverse group of researchers and evaluators as they shift from using narrow accountability instruments to appreciating emergent causal pathways within research for development.

Les grands programmes de recherche financés par des fonds publics continuent de recevoir des investissements accrus en tant qu'interventions visant à produire un impact pour les populations les plus pauvres et les plus marginalisées dans le monde. À cette intersection entre la recherche et le développement, la recherche devrait contribuer aux processus complexes de changement sociétal. Pour adopter une vision coconstruite de l'impact comme phénomène émergeant au fil de liens de causalité incertains, bien souvent sans résultats prédéfinis, il faut innover en utilisant des approches d'évaluation sensibles à la complexité. Les articles de ce numéro spécial présentent de riches expériences d'auteurs travaillant dans différents secteurs et zones géographiques, employant l'innovation méthodologique et le pouvoir de navigation tout en réconciliant les tensions. Ils illustrent les défis lorsqu'il s'agit (i) d'évaluer des performances pour répondre aux exigences de redevabilité tout en favorisant l'apprentissage pour l'adaptation; (ii) d'évaluer les théories prospectives du changement tout en saisissant le changement émergent; (iii) d'évaluer les dimensions relationnelles internes tout en mesurant les résultats de développement externes; (iv) d'évaluer à différentes échelles: de la mesure de l'impact final au niveau local à la compréhension des contributions au changement au niveau des systèmes. Pris dans son ensemble, ce numéro illustre la façon dont l'évaluation de la recherche pour le développement mûrit à travers les expériences d'un groupe toujours plus important et divers de chercheurs et d'évaluateurs qui abandonnent des outils de redevabilité étriqués afin d'apprécier les liens de causalité émergents au sein de la recherche pour le développement.

How are Research for Development Programmes Implementing and Evaluating Equitable Partnerships to Address Power Asymmetries?

The complexity of issues addressed by research for development (R4D) requires collaborations between partners from a range of disciplines and cultural contexts. Power asymmetries within such partnerships may obstruct the fair distribution of resources, responsibilities and benefits across all partners. This paper presents a cross-case analysis of five R4D partnership evaluations, their methods and how they unearthed and addressed power asymmetries. It contributes to the field of R4D partnership evaluations by detailing approaches and methods employed to evaluate these partnerships. Theory-based evaluations deepened understandings of how equitable partnerships contribute to R4D generating impact and centring the relational side of R4D. Participatory approaches that involved all partners in developing and evaluating partnership principles ensured contextually appropriate definitions and a focus on what partners value. Supplementary Information: The online version contains supplementary material available at 10.1057/s41287-023-00578-w.

Finally, centring reflexivity within a learning oriented approach ensured that partnership evaluation findings were used to adapt and improve the way R4D programmes operate. La complexité des enjeux abordés par la recherche pour le développement (R4D) nécessite des collaborations entre des partenaires de disciplines et de contextes culturels variés. Les asymétries de pouvoir au sein d'un tel partenariat peuvent entraver la répartition équitable des ressources, des responsabilités et des avantages entre tous les partenaires. Cet article présente une analyse croisée de cinq évaluations de partenariats R4D, leurs méthodes et la manière dont elles ont mis au jour et traité les asymétries de pouvoir. Cet article contribue au domaine des évaluations de partenariats R4D en décrivant de façon détaillée les approches et les méthodes employées pour évaluer ces partenariats. Les évaluations basées sur la théorie ont permis d'approfondir la compréhension de la manière dont les partenariats équitables contribuent à l'impact de la R4D et à mettre au centre l'aspect relationnel de la R4D. Les approches participatives impliquant tous les partenaires dans l'élaboration et l'évaluation des principes du partenariat ont permis de garantir des définitions adaptées au contexte et de faire en sorte que les évaluations se penchent sur ce qui est jugé important pour tous les partenaires. Enfin, le fait de mettre la réflexivité au cœur de l'analyse dans une approche axée sur l'apprentissage a permis d'utiliser les résultats de l'évaluation du partenariat pour adapter et améliorer le fonctionnement des programmes de R4D.

Revealing the Relational Mechanisms of Research for Development Through Social Network Analysis.

Achieving impact through research for development programmes (R4D) requires engagement with diverse stakeholders across the research, development and policy divides. Understanding how such programmes support the emergence of outcomes, therefore, requires a focus on the relational aspects of engagement and collaboration. Increasingly, evaluation of large research collaborations is employing social network analysis (SNA), making use of its relational view of causation. In this paper, we use three applications of SNA within similar large R4D programmes, through our work within evaluation of three Interidsiplinary Hubs of the Global Challenges Research Fund, to explore its potential as an evaluation method. Our comparative analysis shows that SNA can uncover the structural dimensions of interactions within R4D programmes and enable learning about how networks evolve through time. We reflect on common challenges across the cases including navigating different forms of bias that result from incomplete network data, multiple interpretations across scales, and the challenges of making causal inference and related ethical dilemmas. We conclude with lessons on the methodological and operational dimensions of using SNA within monitoring, evaluation and learning (MEL) systems that aim to support both learning and accountability. Supplementary Information: The online version contains supplementary material available at 10.1057/s41287-023-00576-y.

Pour que les programmes de recherche pour le développement (R4D ou Research for Developmement en anglais) aient un impact, il faut un engagement entre diverses parties prenantes dans les domaines de la recherche, du développement et des politiques. Il est nécessaire de se concentrer sur les aspects relationnels de l'engagement et de la collaboration si l'on souhaite comprendre la façon dont ce type de programme permet l'émergence de résultats. L'évaluation des grands consortia de recherche utilise de plus en plus fréquemment l'analyse des réseaux sociaux (SNA ou social network analysis en anglais) en appliquant sa vision relationnelle de la causalité. Dans cet article, en vue d'explorer son potentiel en tant que méthode d'évaluation, nous utilisons trois applications d'analyse des réseaux sociaux au sein de grands programmes R4D similaires dans le cadre de notre travail d'évaluation de trois pôles interdisciplinaires du Fonds de recherche sur les défis mondiaux. Notre analyse comparative montre que l'analyse des réseaux sociaux peut révéler les dimensions structurelles des interactions au sein de ces programmes et permettre d'apprendre comment les réseaux évoluent dans le temps. Nous menons une réflexion quant aux défis communs qui émanent de ces cas, y compris la gestion de différentes formes de biais qui résultent de données de réseau incomplètes, de multiples interprétations sur des échelles différentes et les défis liés au fait d'établir une inférence causale et les dilemmes éthiques connexes. Nous concluons par des leçons sur les dimensions méthodologiques et opérationnelles de l'utilisation de l'analyse des réseaux sociaux dans les systèmes de suivi, d'évaluation et d'apprentissage (SEA) qui visent à soutenir à la fois l'apprentissage et la redevabilité.

The impact of Indigenous-led programs on alcohol-related criminal incidents: a multiple baseline design evaluation.

OBJECTIVES: To evaluate the impact of a multi-component, Aboriginal-led strategy to reduce alcohol-related criminal incidents for Aboriginal people in four rural/remote communities in New South Wales (NSW), Australia. METHODS: A retrospective multiple baseline design (MBD), using interrupted time series analysis of routinely collected crime data. RESULTS: A statistically significant reduction in alcohol-related criminal incidents was observed in one community for both victims of crime (parameter estimate -0.195; p≤0.01) and persons of interest (parameter estimate -0.282; p≤0.001). None of the analyses show level shifts, meaning there were no measurable changes immediately post the introduction of the Breaking the Cycle (BTC) programs. CONCLUSIONS: It is not possible to conclude that the program was effective independently of any other community factors, because the statistically significant result was not observed across multiple communities. The statistically significant result in one community has clear practical benefits in that community: a sustained impact over two years would reduce Aboriginal victims of alcohol-related crime from an estimated 56 incidents per annum to 36, and reduce Aboriginal persons of interest in alcohol-related crime from an estimated 68 alcohol-related person of interest (POI) per annum to 40. IMPLICATIONS FOR PUBLIC HEALTH: The statistically and practically meaningful result in Community 1 highlights the potential of multi-component, Aboriginal-led strategies to reduce alcohol-related criminal incidents. Earlier engagement with researchers, to identify best-evidence strategies to reduce alcohol harms and to facilitate the use of prospective evaluation designs, would help translate the positive outcome in one community across multiple communities.

A Web-Based Alcohol and Other Drug Prevention Program (Strong & Deadly Futures) for Aboriginal and Torres Strait Islander School Students: Protocol for a Cluster Randomized Controlled Trial.

BACKGROUND: There are no available school-based alcohol and drug prevention programs with evidence of effectiveness among Aboriginal and Torres Strait Islander youth. To address this, we codeveloped the Strong & Deadly Futures well-being and alcohol and drug prevention program in partnership with an Indigenous creative design agency and 4 Australian schools. OBJECTIVE: This paper presents the protocol to evaluate the effectiveness of Strong & Deadly Futures in reducing alcohol and other drug use and improving well-being among Aboriginal and Torres Strait Islander youth. METHODS: The target sample will be 960 year 7 and 8 students from 24 secondary schools in Australia, of which approximately 40% (384/960) will identify as Aboriginal or Torres Strait Islander. The study design is a 2-group, parallel cluster randomized controlled trial with allocation concealment. Recruited schools will be block randomized (ratio 1:1), stratified by geographical remoteness, by an independent statistician. Schools will be randomized to receive Strong & Deadly Futures, a web-based alcohol and drug prevention and social and emotional well-being program that delivers curriculum-aligned content over 6 lessons via an illustrated story, or health education as usual (control). Control schools will be supported to implement Strong & Deadly Futures following trial completion. Surveys will be administered at baseline, 6 weeks, 12 months, and 24 months (primary end point) post baseline. Primary outcomes are alcohol use (adapted from the National Drug Strategy Household Survey), tobacco use (Standard High School Youth Risk Behavior Survey), and psychological distress (Kessler-5 Psychological Distress Scale). Secondary outcomes are alcohol and drug knowledge and intentions, alcohol-related harms, binge drinking, cannabis use, well-being, empowerment, appreciation of cultural diversity, and truancy. RESULTS: The trial was funded by the National Health and Medical Research Council in January 2019, approved by the Human Research Ethics Committee of the University of Sydney (2020/039, April 2020), the Aboriginal Health and Medical Research Council of New South Wales (1620/19, February 2020), the Western Australian Aboriginal Health Ethics Committee (998, October 2021), and the ethics committees of each participating school, including the New South Wales Department of Education (2020170, June 2020), Catholic Education Western Australia (RP2020/39, November 2020), and the Queensland Department of Education (550/27/2390, August 2021). Projected dates of data collection are 2022-2024, and we expect to publish the results in 2025. A total of 24 schools have been recruited as of submission of the manuscript. CONCLUSIONS: This will be the first cluster randomized controlled trial of a culturally inclusive, school-based alcohol and drug prevention program for Aboriginal and Torres Strait Islander youth; therefore, it has significant potential to address alcohol and other drug harms among Aboriginal and Torres Strait Islander youth. TRIAL REGISTRATION: Australian New Zealand Clinical Trials Registry ACTRN12620001038987; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=380038&isReview=true. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/34530.

Strong and Deadly Futures: Co-Development of a Web-Based Wellbeing and Substance Use Prevention Program for Aboriginal and Torres Strait Islander and Non-Aboriginal Adolescents.

School-based programs can effectively prevent substance use; however, systematic reviews and consultation with stakeholders identified a need for effective, culturally inclusive programs for Aboriginal and/or Torres Strait Islander (hereafter Aboriginal) youth. This paper describes the development of Strong & Deadly Futures, a six-lesson, curriculum-aligned wellbeing and substance use prevention program that was designed for, and with, the Aboriginal youth. Formative reviews and consultation recommended that the program (i) combine effective components of mainstream prevention with cultural elements, highlighting Aboriginal cultural strengths; (ii) avoid stigma and celebrates the cultural diversity by catering to both Aboriginal and non-Aboriginal students; and (iii) use digital technology to enhance engagement, implementation and scalability. Guided by an Appreciative Inquiry approach, the program was developed in partnership with an Indigenous Creative Design Agency, and four schools in New South Wales and Queensland, Australia. Aboriginal (n = 41) and non-Aboriginal students (n = 36) described their role models, positive aspects of their community and reasons to avoid substance use; these formed the basis of an illustrated story which conveyed the key learning outcomes. Feedback from teachers, students and content experts supported the acceptability of the program, which will be evaluated in a subsequent randomised controlled trial.

An ecological model of drug and alcohol use and related harms among Aboriginal and Torres Strait Islander Australians: A systematic review of the literature.

Globally, Indigenous populations experience a disproportionately higher burden of disease related to substance use. Effective prevention of harm related to substance use is a key strategy for improving the health and wellbeing of Aboriginal and Torres Strait Islander peoples in Australia. To inform preventative approaches, this review synthesised the evidence of risk and protective factors of substance use and related harms among Aboriginal and Torres Strait Islander peoples. Eight peer-reviewed and two grey literature databases were systematically searched for quantitative or qualitative studies assessing factors associated with substance use and related harms among Aboriginal and Torres Strait Islander peoples, published between 1 January 1990 and 30 April 2018. Study quality was assessed using validated instruments. Risk or odds ratios were extracted or calculated and factors were summarised in an ecological model into individual, relationship, community, societal or culturally-distinct levels. Thirty-eight relevant studies were identified and reviewed. Individual-level risk factors for substance use were identified including low socio-economic status, high psychological distress, poly drug use and being male. Relationship-level factors were peer pressure and partner/family substance use; protective factors were supportive environments and positive role models. Community-level risk factors included availability of substances. Culturally-distinct factors included cultural connection as a protective factor, but cultural obligations around sharing was a risk factor. Societal risk factors included intergenerational trauma caused by government policies. These findings highlight the importance of tailored preventative approaches for Aboriginal and Torres Strait Islander communities that address identified risk factors and promote protective factors across all ecological levels.

'We walked side by side through the whole thing': A mixed-methods study of key elements of community-based participatory research partnerships between rural Aboriginal communities and researchers.

OBJECTIVES: To advance the rural practice in working with Aboriginal communities by (a) identifying the extent of community partners' participation in and (b) operationalising the key elements of three community-based participatory research partnerships between university-based researchers and Australian rural Aboriginal communities. DESIGN: A mixed-methods study. Quantitative survey and qualitative one-on-one interviews with local project implementation committee members and group interviews with other community partners and project documentation. SETTING: Three rural Aboriginal communities in New South Wales. PARTICIPANTS: Thirty-seven community partners in three community-based participatory research partnerships of which 22 were members of local project implementation committees and 15 were other community partners who implemented activities. INTERVENTION: Community-based participatory research partnerships to develop, implement and evaluate community-based responses to alcohol-related harms. MAIN OUTCOMES MEASURES: Community partners' extent of and experiences with participation in the community-based participatory research partnership and their involvement in the development and implementation processes. RESULTS: Community partners' participation varied between communities and between project phases within communities. Contributing to the community-based participatory research partnerships were four key elements of the participatory process: unique expertise of researchers and community-based partners, openness to learn from each other, trust and community leadership. CONCLUSION: To advance the research practice in rural Aboriginal communities, equitable partnerships between Aboriginal community and research partners are encouraged to embrace the unique expertise of the partners, encourage co-learning and implement community leadership to build trust.

Factors Associated with Alcohol-Related Injuries for Aboriginal and Non-Aboriginal Australians: An Observational Study.

Alcohol use and related injuries are a leading risk factor for deaths and disabilities in Australia, particularly for Aboriginal and Torres Strait Islander people. An improved understanding of individual and geographical community characteristics that are significantly associated with higher rates of alcohol-related injuries for specific populations can contribute to more effective efforts aimed at reducing alcohol-related injuries. For Aboriginal and non-Aboriginal Australians in New South Wales, this study used emergency department (ED) data to investigate rates of alcohol-related injuries, whether differences in rates vary between communities, and individual and community characteristics significantly associated with alcohol-related injuries. Differences in rates of alcohol-related injuries between Aboriginal and non-Aboriginal people varied significantly between communities. Being younger than 38 years old was significantly associated with increased risk of alcohol-related injuries, independent of Aboriginal status and gender. Increased disadvantage of the geographical community inhabited was associated with increased alcohol-related injuries for males. For Aboriginal males, living in a regional community was significantly associated with increased alcohol-related injuries, compared to living in major cities. Conversely, for non-Aboriginal people, living in regional communities was significantly associated with fewer alcohol-related injuries. It is therefore likely that an explanation for between-community differences can be found in regional communities.

Preventing Substance Use Among Indigenous Adolescents in the USA, Canada, Australia and New Zealand: a Systematic Review of the Literature.

This systematic review assessed the current evidence base of substance use prevention programs for Indigenous adolescents in the USA, Canada, Australia and New Zealand. The authors investigated (a) the outcomes, type, setting and context of prevention programs; (b) the common components of beneficial prevention programs; and (c) the methodological quality of evaluations of included prevention programs. The authors searched eight peer-reviewed and 20 grey literature databases for studies published between 1 January 1990 and 31 August 2017. Data extracted included type of program (culturally adapted, culture-based or unadapted), the setting (school, community, family or multi-setting), delivery (computerised or traditional), context (Indigenous-specific or multi-cultural environment) and common components of the programs. Program evaluation methodologies were critically appraised against standardised criteria. This review identified 26 eligible studies. Substance use prevention programs for Indigenous youth led to reductions in substance use frequency and intention to use; improvements in substance-related knowledge, attitudes and resistance strategies; and delay in substance use initiation. Key elements of beneficial programs included substance use education, skills development, cultural knowledge enhancement and community involvement in program development. Five programs were rated as methodologically strong, seven were moderate and fourteen were weak. Prevention programs have the potential to reduce substance use among Indigenous adolescents, especially when they are developed in partnership with Indigenous people. However, more rigorously conducted evaluation trials are required to strengthen the evidence base.

Developing an ecological framework of factors associated with substance use and related harms among Aboriginal and Torres Strait Islander people: protocol for a systematic review.

INTRODUCTION: Aboriginal and Torres Strait Islander people experience high rates of substance use and related harms. Previous prevention programmes and policies have met with limited success, particularly among youth, and this may be a result of inadequately targeting the unique risk and protective factors associated with substance use for Aboriginal and Torres Strait Islander people. The purpose of this systematic review is to therefore synthesise the risk and protective factors associated with substance use and related harms among Aboriginal and Torres Strait Islander people, and critically appraise the methodological quality of the included studies. METHODS AND ANALYSIS: A total of seven peer-reviewed (Cochrane, Embase, PsychInfo, Medline, ProQuest, Informit, and CINAHL) and two grey literature (Heath InfoNet and Closing the Gap Clearinghouse) databases will be systematically searched using search terms in line with the aims of this review and based on previous relevant reviews. Studies published between 1 January 1990 and 31 April 2018 will be included if they identify risk and/or protective factors for substance use or related harms in a study sample that consists of at least 50% Aboriginal and Torres Strait Islander people. A narrative synthesis will be undertaken where the identified factors will be organised using an ecological approach into individual, relationship, community, societal and cultural levels. A critical appraisal of study quality will be conducted using the Joanna Briggs Institute Critical Appraisal Checklist for Studies Reporting Prevalence Data and the qualitative assessment tool by Godfrey and Long. ETHICS AND DISSEMINATION: Formal ethics approval is not required as primary data will not be collected. The results will be disseminated through a peer-reviewed publication, conference presentations and social media. PROSPERO REGISTRATION NUMBER: CRD42017073734.

A systematic review of evaluations of prison-based alcohol and other drug use behavioural treatment for men.

OBJECTIVE: A history of alcohol and other drug (AoD) use is common among men entering prison and often linked to the crime for which they are imprisoned. This is the first systematic review of prison-based, behavioural AoD treatment programs for more than a decade and the first that reviews the methodological quality of evaluations. This review aims to create an understanding of the quality of research in this field and identify the most effective AoD use treatment for men in prison. METHODS: A PRISMA-compliant systematic review of international, peer-reviewed research published between January 1995 and December 2015. The Dictionary for Effective Public Health Practice Project was used to assess the methodological quality of papers. RESULTS: A total of 25 relevant papers were identified, of which 12 were rated as methodologically sound. Four of these measured post-release AoD use and three reported statistically significant reductions in AoD use. CONCLUSIONS: Although there is relatively little methodologically strong evidence of the impact of prison-based AoD treatment, and no Australian papers studies, current best-evidence practice is Cognitive behavioural therapy delivered in Therapeutic Community (TC) settings. Implications for public health: Prison-based TC treatment should be available to people in prison who have a history of AoD use.

A Need for Tailored Programs and Policies to Reduce Rates of Alcohol-related Crimes for Vulnerable Communities and Young People: An Analysis of Routinely Collected Police Data.

BACKGROUND AND AIMS: Given ongoing community concern about high rates of alcohol-related crimes (ARCs) experienced by disadvantaged populations, a more specific and nuanced understanding of factors associated with ARCs would help inform the development of more sophisticated programs and policies aimed at reducing ARCs. This study estimates rates of ARCs across all communities in New South Wales (NSW), Australia, using routinely collected police data; investigates whether there are differences between communities; and identifies individual and community characteristics that are significantly associated with higher rates of ARCs. SHORT SUMMARY: This study analysed routinely collected police data in New South Wales, Australia, to identify individual and community characteristics associated with alcohol-related crimes. Young people, Aboriginal Australians, socio-economically disadvantaged communities, remote and regional communities and communities with higher per capita rate of on-venue liquor licenses are at risk of alcohol-related crimes. METHODS: Age standardized rates of ARCs were calculated. A multi-level Poisson regression analysis was conducted to investigate the individual and community factors that were statistically significantly associated with higher rates of ARC, separately for Aboriginal and non-Aboriginal Australians. RESULTS: Rates of ARCs were statistically significantly higher for Aboriginal Australians, young people (aged 13-37 years) and on weekends. ARCs varied significantly across communities, and were significantly higher in remote or regional communities, in communities with a higher per capita rate of on-venue licences, and for socio-economically disadvantaged communities for non-Aboriginal Australians, but not for Aboriginal females. CONCLUSION: This analysis shows that the impact of national-level and jurisdictional-level legislation and policies is uneven across communities and defined populations, leaving young people, socio-economically disadvantaged communities and Aboriginal Australians at increased risk of ARCs. To more equitably reduce the exposure of all Australians to ARC, mechanisms that effectively engage vulnerable communities and defined populations, need to be developed in consultation with them, implemented and evaluated.

Substance Use Prevention Programs for Indigenous Adolescents in the United States of America, Canada, Australia and New Zealand: Protocol for a Systematic Review.

BACKGROUND: Indigenous adolescents are at a higher risk of experiencing harms related to substance use compared with their non-Indigenous counterparts as a consequence of earlier onset and higher rates of substance use. Early onset of substance use has been identified as a risk factor for future substance use problems and other health, social, and family outcomes. Therefore, prevention of substance use among adolescents has been identified as a key area to improve health of Indigenous Peoples. Evidence exists for the effectiveness of prevention approaches for adolescents in mainstream populations and, most recently, for the use of computer- and Internet-delivered interventions to overcome barriers to implementation. However, there is currently no conclusive evidence about the effectiveness of these approaches for Indigenous adolescents. OBJECTIVE: The purpose of this review is to synthesize the international evidence regarding the effectiveness of substance use prevention programs for Indigenous adolescents in the United States, Canada, Australia, and New Zealand. METHODS: A total of 8 peer-reviewed databases and 20 gray literature databases will be searched, using search terms in line with the aims of this review and based on previous relevant reviews of substance use prevention. Studies will be included if they evaluate a substance use prevention program with Indigenous adolescents (aged 10 to 19 years) as the primary participant group and are published between January 1, 1990 and August 31, 2017. RESULTS: A narrative synthesis will be provided about the effectiveness of the programs, the type of program (whether culture-based, adapted, or unadapted), delivery of the program (computer- and Internet-delivered or traditional), and the setting in which the programs are delivered (community, school, family, clinical, or a combination). CONCLUSIONS: The study will identify core elements of effective substance use prevention programs among Indigenous adolescents and appraise the methodological quality of the studies. This review will provide researchers, policy makers, and program developers with evidence about the potential use of prevention approaches for Indigenous adolescents.

Riding the rural radio wave: The impact of a community-led drug and alcohol radio advertising campaign in a remote Australian Aboriginal community.

OBJECTIVE: Aboriginal people experience a higher burden of disease as a consequence of drug and alcohol (D&A) abuse. Although media campaigns can be a popular tool for disseminating health promotion messages, evidence of the extent to which they reduce the impact of substance abuse is limited, especially for rural Aboriginal communities. This paper is the first to examine the impact a locally designed D&A radio advertising campaign for Aboriginal people in a remote community in Western NSW. DESIGN: A post-intervention evaluation. SETTING: The radio campaign was implemented in Bourke, (population 2465, 30% Aboriginal). PARTICIPANTS: Fifty-three community surveys were completed. MAIN OUTCOME MEASURE(S): The self-reported level of awareness of the campaign and the number of self-referrals to local D&A workers in the intervention period. RESULTS: Most respondents (79%) reported they listen to radio on a daily basis, with 75% reporting that they had heard one or more of the advertisements. The advertisement that was remembered best contained the voice of a respected, local person. There was one self-referral to local health services during the intervention timeframe. CONCLUSION: The community-led radio advertising campaign increased community awareness of substance abuse harms, but had limited impact on formal help-seeking. This paper highlights the value of radio as a commonly used, trusted and culturally relevant health promotion medium for rural communities, especially when engaging local respected Aboriginal presenters.

The Feasibility of Embedding Data Collection into the Routine Service Delivery of a Multi-Component Program for High-Risk Young People.

BACKGROUND: There is little evidence about how to improve outcomes for high-risk young people, of whom Indigenous young people are disproportionately represented, due to few evaluation studies of interventions. One way to increase the evidence is to have researchers and service providers collaborate to embed evaluation into the routine delivery of services, so program delivery and evaluation occur simultaneously. This study aims to demonstrate the feasibility of integrating best-evidence measures into the routine data collection processes of a service for high-risk young people, and identify the number and nature of risk factors experienced by participants. METHODS: The youth service is a rural based NGO comprised of multiple program components: (i) engagement activities; (ii) case management; (iii) diversionary activities; (iv) personal development; and (v) learning and skills. A best-evidence assessment tool was developed by staff and researchers and embedded into the service's existing intake procedure. Assessment items were organised into demographic characteristics and four domains of risk: education and employment; health and wellbeing; substance use; and crime. Descriptive data are presented and summary risk variables were created for each domain of risk. A count of these summary variables represented the number of co-occurring risks experienced by each participant. The feasibility of this process was determined by the proportion of participants who completed the intake assessment and provided research consent. RESULTS: This study shows 85% of participants completed the assessment tool demonstrating that data on participant risk factors can feasibly be collected by embedding a best-evidence assessment tool into the routine data collection processes of a service. The most prevalent risk factors were school absence, unemployment, suicide ideation, mental distress, substance use, low levels of physical activity, low health service utilisation, and involvement in crime or with the juvenile justice system. All but one participant experienced at least two co-occurring domains of risk, and the majority of participants (58%) experienced co-occurring risk across four domains. CONCLUSIONS: This is the first study to demonstrate that best-evidence measures can feasibly be embedded into the routine data collection processes of a service for high-risk young people. This process allows services to tailor their activities to the most prevalent risks experienced by participants, and monitor these risks over time. Replication of this process in other services would improve the quality of services, facilitate more high quality evaluations of services, and contribute evidence on how to improve outcomes for high-risk young people.

A systematic review of studies evaluating Australian indigenous community development projects: the extent of community participation, their methodological quality and their outcomes.

BACKGROUND: Community development is a health promotion approach identified as having great potential to improve Indigenous health, because of its potential for extensive community participation. There has been no systematic examination of the extent of community participation in community development projects and little analysis of their effectiveness. This systematic review aims to identify the extent of community participation in community development projects implemented in Australian Indigenous communities, critically appraise the qualitative and quantitative methods used in their evaluation, and summarise their outcomes. METHODS: Ten electronic peer-reviewed databases and two electronic grey literature databases were searched for relevant studies published between 1990 and 2015. The level of community participation and the methodological quality of the qualitative and quantitative components of the studies were assessed against standardised criteria. RESULTS: Thirty one evaluation studies of community development projects were identified. Community participation varied between different phases of project development, generally high during project implementation, but low during the evaluation phase. For the majority of studies, methodological quality was low and the methods were poorly described. Although positive qualitative or quantitative outcomes were reported in all studies, only two studies reported statistically significant outcomes. DISCUSSION: Partnerships between researchers, community members and service providers have great potential to improve methodological quality and community participation when research skills and community knowledge are integrated to design, implement and evaluate community development projects. CONCLUSION: The methodological quality of studies evaluating Australian Indigenous community development projects is currently too weak to confidently determine the cost-effectiveness of community development projects in improving the health and wellbeing of Indigenous Australians. Higher quality studies evaluating community development projects would strengthen the evidence base.