An assessment of program evaluation methods and quality in Australian prevention agencies.

ISSUE ADDRESSED: This study aimed to examine evaluation methods and quality in Australian health promotion agencies and the factors associated with this. The evidence base for prevention strategies is limited, with the evidence generated through program evaluation by health promotion and disease prevention agencies lacking rigour. Despite the need to improve the quality of evaluation, there is limited evidence of what influences evaluation quality in the prevention field. METHODS: Data were collected using the Evaluation Practice Analysis Survey and an audit and appraisal of evaluation reports. Descriptive analysis was used to examine evaluation characteristics and multivariable regression was used to explore the association between evaluation and organisational attributes and evaluation quality. RESULTS: In total, 392 evaluation reports were reviewed from 78 government and non-government agencies. Process evaluation was conducted most frequently, followed by impact evaluation. Overall evaluation quality was low (median 24.5%). In multivariable regression analysis, only two factors were associated with evaluation quality: health promotion budget (ratio of geometric means 1.53 [95% CI 1.02-2.29]); and, conducting statewide or national prevention programs (1.38 [95% CI 1.05-1.82]). CONCLUSIONS: The findings show that the potential to improve evaluation quality is greatest in smaller organisations that deliver health promotion at a local or regional scale. SO WHAT?: By improving the rigour of existing evaluation, there is opportunity to build the evidence base for prevention strategies, which highlights the importance of embedding the enablers of program learning and evidence generation within health promotion and prevention organisations.

Barriers and opportunities in the translation of mobile phone and social media interventions between research and health promotion practice in Australia: a qualitative study of expert perspectives.

BACKGROUND: Newer technologies, such as smartphones and social networking sites, offer new opportunities for health promotion interventions. There is evidence to show that these technologies can be effectively and acceptably used for health promotion activities. However, most interventions produced in research do not end up benefitting non-research populations, while the majority of technology-facilitated interventions which are available outside of research settings are either undocumented or have limited or no evidence to support any benefit. We therefore aimed to explore the perspectives of researchers and health promotion experts on efforts to translate technology-facilitated prevention initiatives into practice, and the barriers to achieving translation. METHODS: We utilised a qualitative study design, involving in-depth interviews with researchers experienced with technology-facilitated prevention interventions and prominent health promotion experts. RESULTS: Some barriers mirror the findings of other studies into health promotion practice, which have found that competing priorities, resource limitations and organisational capacity are important in determining use of evidence in programme planning, engagement in translation and evaluation practice. We add to this literature by describing barriers that are more specifically related to technology-facilitated prevention, such as the pace of developments in technology, and how this clashes with the time taken to develop and ready evidence for translation. CONCLUSIONS: In order to maximise the vast potential of technology-facilitated prevention interventions to promote population health, it is essential that translation is at the forefront of consideration for both researchers and practitioners. We suggest actions that can be taken by both researchers and practitioners to improve translation of technology-facilitated prevention interventions, and also highlight how funding schemes can be modified to facilitate translation.

Facilitators of Attendance and Adherence to Group-Based Physical Activity for Older Adults: A Literature Synthesis.

This review examines program features that influence attendance and adherence to group-based physical activity (PA) by older adults. Medline, PubMed, CINAHL plus, PsycINFO, and the Cochrane Library were searched for studies published from 1995-2016. Quantitative and qualitative studies investigating factors related to PA group attendance or adherence by persons aged 55 years and over were included. Searching yielded eight quantitative and 13 qualitative studies, from 2,044 titles. Quantitative findings identified social factors, instructor characteristics, PA types, class duration and frequency, and perceived PA outcomes as important for attendance and adherence, whilst qualitative studies identified settings, leadership, PA types, observable benefits, and social support factors. Studies were predominantly low- to moderate-quality. This review identified design and delivery considerations for group-based PA programs to inform best-practice frameworks and industry capacity building. Future research should use longitudinal and mixed-methods designs to strengthen evidence about facilitators of program reach and engagement.

Factors influencing care and support for older adults with traumatic injury in Australia: a qualitative study.

PURPOSE: To investigate factors influencing provision of care and support to older adults with traumatic injury in Australia, from a health service and policy perspective. METHODS: Semi-structured interviews were undertaken with 16 clinicians, support providers, researchers, policy makers, and representatives from peak bodies in Australia, who had experience across injury, ageing, and disability sectors. A thematic analysis was performed using a framework approach. RESULTS: Themes identified included prolonged injury recovery in older adults, limited accommodation options, restricted access to preventive care, escalating care needs over time, issues with siloed funding schemes and funding availability, and the need for advocacy to coordinate care across different schemes. Certain themes were specific to people who were older when injured. Others related to people who had acquired an injury at a younger age and were ageing with injury. However, most themes had relevance for both groups. CONCLUSIONS: For older adults, this research has highlighted a range of cross-sector problems which impact upon the potential to recover from injury and to age well with injury. To improve the lives of older adults with injuries, Australia needs better coordinated system interfaces, shared funding models or packages of care across sectors, and improved advocacy and case management.Implications for rehabilitationTo address the lack of long-term, multi-disciplinary preventive care for age-related conditions and secondary complications in people with injuries, older adults may need more prolonged, or individualised, care following injury, episodic health checks, and a greater focus on long-term health care.Accessible and well-coordinated specialist housing and support responses, that offer timely access to health care professionals and carers trained in both ageing and disability, and enable greater support for "ageing in place", are required to manage changing care needs of people ageing with injury.There needs to be greater involvement of geriatricians, nurses, and allied health care professionals within the aged care sector in order to care for people with complex needs, including older adults with injury-related disability.To reduce gaps in care and support for injured older adults, Australia needs better coordinated system interfaces, shared funding models or packages of care across sectors.

How practitioner, organisational and system-level factors act to influence health promotion evaluation capacity: Validation of a conceptual framework.

The need to improve the practice and quality of evaluation in the health promotion and disease prevention field is widely recognised. In order to plan, implement and evaluate health promotion evaluation capacity building efforts, there is a need to better understand the practitioner, organisational and system-level determinants of evaluation capacity and practice. This study aimed to assess the validity Evaluation Practice Analysis Survey (EPAS) constructs using confirmatory factor analysis and validate a conceptual framework of health promotion evaluation capacity using path analysis. Experienced Australian health promotion practitioners completed the survey (n = 219). Twenty-one of the original 23 EPAS scales were assessed as reliable and valid. The final model was found to have good fit (χ214 = 18.72, p = 0.18, root mean square error of approximation = 0.04, 90% CI 0.00-0.82, Comparative Fit Index = 1.00, standardised root mean square residual = 0.04). This model supports the role of the organisation in facilitating evaluation practice through leadership, culture, systems, support and resources. It builds on existing frameworks from other fields to incorporate political, funding and administrative factors. This study provides an evidence-based model of evaluation capacity that organisations, funders and policy makers can use to plan and implement more effective evaluation capacity building strategies within organisations and the wider prevention field.

Understanding the factors that influence health promotion evaluation: The development and validation of the evaluation practice analysis survey.

The demand for improved quality of health promotion evaluation and greater capacity to undertake evaluation is growing, yet evidence of the challenges and facilitators to evaluation practice within the health promotion field is lacking. A limited number of evaluation capacity measurement instruments have been validated in government or non-government organisations (NGO), however there is no instrument designed for health promotion organisations. This study aimed to develop and validate an Evaluation Practice Analysis Survey (EPAS) to examine evaluation practices in health promotion organisations. Qualitative interviews, existing frameworks and instruments informed the survey development. Health promotion practitioners from government agencies and NGOs completed the survey (n = 169). Principal components analysis was used to determine scale structure and Cronbach's α used to estimate internal reliability. Logistic regression was conducted to assess predictive validity of selected EPAS scale. The final survey instrument included 25 scales (125 items). The EPAS demonstrated good internal reliability (α > 0.7) for 23 scales. Dedicated resources and time for evaluation, leadership, organisational culture and internal support for evaluation showed promising predictive validity. The EPAS can be used to describe elements of evaluation capacity at the individual, organisational and system levels and to guide initiatives to improve evaluation practice in health promotion organisations.