Systematic Screening and Assessment of Hospital-Based Youth Violence Prevention Programs.

Youth violence is a preventable public health issue. Few hospital-based programs intentionally focus on youth violence prevention. This project aimed to describe the Systematic Screening and Assessment (SSA) methodology used to identify existing hospital-based youth violence prevention (HBYVP) programs ready for future rigorous evaluation. To identify promising HBYVP programs currently in use and assess readiness for evaluation, data from the 2017 American Hospital Association (AHA) Annual Survey of Hospitals was used to identify hospitals with Level I-III trauma centers with reported HBYVP programs. Information for each program was gathered via environmental scan and key informant interviews. A total of 383 hospital-based violence prevention programs were identified. Two review panels were conducted with violence prevention experts to identify characteristics of programs suitable for an evaluability assessment (EA). Fifteen programs focused on youth (10-24 years old) and were identified to be promising and evaluable. Three of the 15 programs were determined to have the infrastructure and readiness necessary for rigorous evaluation. Lessons learned and best practices for SSA project success included use of streamlined outreach efforts that provide program staff with informative and culturally tailored project materials outlining information about the problem, project goals, proposed SSA activities, and altruistic benefit to the community at the initial point of contact. In addition, success of review panels was attributed to use of software to streamline panelist review processes and use of evaluation and data analysis subject matter experts to serve as panel facilitators. Communities experiencing high youth violence burden and hospitals serving these communities can improve health outcomes among youth by implementing and evaluating tailored HBYVP programs.

Study protocol for an evaluability assessment of an anti-human trafficking program.

BACKGROUND: Human trafficking is a serious global challenge associated with a complex array of health inequities for individuals, families, and communities. Consequently, in addition to a conventional criminal justice approach, anti-trafficking scholars have increasingly called for a public health approach to address this global challenge. Such calls have emphasized that a comprehensive, robust, and social justice-informed public health strategy for anti-trafficking must include services to facilitate survivors' HT exit and recovery, and to prevent their re-victimization. Fortunately, many community-based organizations and non-governmental organizations worldwide have heeded these calls and developed anti-trafficking programs for survivors. Unfortunately, despite the growing numbers of organizations providing anti-trafficking services, research concerning these programs' effectiveness remains nascent overall, and even more scant when filtered through an equity focus. METHODS: To advance the field by developing guidance concerning how best to evaluate anti-human trafficking programs, an ongoing research project aims to conduct a mixed methods evaluability assessment of a prominent anti-trafficking program using a social justice framework. Guided by well-established evaluability assessment frameworks, the study activities include four sequential steps: (a) focusing the assessment, (b) developing the program theory and logic, (c) gathering feedback, and (d) applying the assessment findings. Activities will include qualitative interviews and focus groups, observations, and quantitative analysis of program data among others. Human subjects and ethical review for the evaluability assessment has been granted by the Office of Human Subjects Research at The University of North Carolina at Chapel Hill. DISCUSSION: Once completed, evaluability assessment results will provide evidence and products that have the potential to guide both evaluation research and service provision not only for the specific organization under study, but also for other anti-human trafficking programs worldwide. Findings will be developed into a variety of dissemination products tailored for both practice professionals and researchers. In the interim, this protocol manuscript offers research strategies and recommendations that can help inform the development of other studies in the developing field of anti-trafficking program evaluation research.

To scope or not to scope? The benefits and challenges of integrating scoping studies in rapid qualitative research and evaluation.

Creating sustainable change and fostering collaborative relationships between researchers and stakeholders is a recognized challenge in the field of evaluation. Identifying programme purpose, cultural context, potential challenges, and engaging stakeholders before an evaluation can produce responsive and impactful evaluations. This paper discusses implementing a targeted scoping study within the framework of rapid qualitative research and evaluation. A scoping study enables collaborative decision-making on evaluation priorities, and functions as an evaluability assessment in time-sensitive contexts. In our experience, a scoping study can be carried out in as little as five days or as long as six weeks. It is timely to revisit the question of what factors influence evaluation outcomes, a scoping study can be used to support an evaluation, address access to data and research, and strengthen communication channels. The methodological approach was used to co-produce an evaluation with an NGO that accurately reflected their needs, recognizing possible challenges and solutions.

The theoretical and practical difficulties of evaluating a community-based 'whole systems' obesity prevention intervention: a research team's critical reflection.

AIMS: This article critically discusses the purpose, pragmatics and politics of conducting commissioned evaluations on behalf of public sector organisations by drawing on the experience of evaluating a community-based 'whole systems' obesity prevention intervention for an English local council. METHODS: The study presented in this article incorporated two approaches: an evaluability assessment that interrogated the theoretical and practical difficulties of evaluating the intervention in a non-political way, and a retrospective analysis using Soft Systems Methodology that interrogated the more political difficulties of conducting such an evaluation in the 'real world'. The information and insights that enabled these reflections came from over 3 years of working closely with the programme team, attending and participating in stakeholder events and meetings, presenting to the Council's Scrutiny Committee meetings, four interviews with the programme manager, and multiple face-to-face group meetings, email exchanges and telephone conversations. RESULTS: The study reveals and analyses three key inter-related challenges that arose during the evaluation of the 'whole systems' obesity prevention intervention: the programme's evaluability, the evaluation purpose, and the nature, role and quality of evidence. CONCLUSIONS: The evaluability assessment was important for defining the programme's theoretical and practical evaluability, and the retrospective analysis using Soft Systems Methodology enabled a greater understanding of the political tensions that existed. Key learning points related to the challenges that arose during this evaluation have broad applicability.

Evaluability assessments as an approach to examining social prescribing.

We report on two evaluability assessments (EAs) of social prescribing (SP) services in South East England conducted in 2016/7. We aimed to demonstrate how EAs can be used to assess whether a programme is ready to be evaluated for outcomes, what changes would be needed to do so and whether the evaluation would contribute to improved programme performance. We also aimed to draw out the lessons learned through the EA process and consider how these can inform the design and evaluation of SP schemes. EAs followed the steps described by Wholey, New Dir Eval 33:77, (1987) and Leviton et al., Annu Rev Public Health 31:213, (2010), including collaboration with stakeholders, elaboration, testing and refinement of an agreed programme theory, understanding the programme reality, identification and review of existing data sources and assessment against key criteria. As a result, evaluation of the services was not recommended. Necessary changes to allow for future evaluation include gaining access to electronic patient records, establishing procedures for collection of baseline and outcome data and linking to data on use of other healthcare services. Lessons learned include ensuring that: (a) SP schemes are developed with involvement (and buy in) of relevant stakeholders; (b) information governance and data sharing agreements are in place from the start; (c) staffing levels are sufficient to cover the range of activities involved in service delivery, data monitoring, reporting, evaluation and communication with stakeholders; (d) SP schemes are co-located with primary care services; and (e) referral pathways and linkages to health service data systems are established as part of the programme design. We conclude that EA provides a valuable tool for informing the design and evaluation of SP schemes. EA can help commissioners to make best use of limited evaluation resources and prioritise which programmes need to be evaluated, as well as how, why and when.

Evaluability assessment of a community development leisure program in Spain.

This paper presents the results of an evaluability assessment (EA) conducted in Spain. EAs are assessments that minimize the uncertainty and risk inherent in a subsequent evaluation (e.g. an impact or results assessment) that may require excessive resources. We designed our own EA model by combining three theoretical approaches, with a holistic and pedagogic vision of the process and a flexible methodology to enable adaptation to different programs and contexts, thus fostering transferability. The EA was carried using a qualitative methodology: documentary analysis, seven semi-structured interviews with stakeholders, eight direct observation sessions and a check-list for politicians and experts. The outcome report helped promote organizational learning, provided the program with a logical and assessable theoretical model, detected its strengths and helped overcome the barriers to efficient assessment. Based on these achievements, an evaluation of the results was recommended, with repeated pretest-posttest measurements and an equivalent control group.

Evaluability assessment: An application in a complex community improvement setting.

Evaluation is essential to understand whether and how policies and other interventions work, why they sometimes fail, and whether they represent a good use of resources. Evaluability assessment (EA) is a means of collaboratively planning and designing evaluations, seeking to ensure they generate relevant and robust evidence that supports decision-making and contributes to the wider evidence base. This article reports on the context, the process undertaken and evidence from participants in an EA facilitated with public service workers involved in implementing a complex, area-based community improvement initiative. This is a novel context in which to conduct an EA. We show how the process allows practitioners at all levels to identify activities for evaluation and co-produce the theory of change developed through the EA. This enables evaluation recommendations to be developed that are relevant to the implementation of the programme, and which take account of available data and resources for evaluation.

Strengthening Evaluation Research: A Case Study of an Evaluability Assessment Conducted in a Carceral Setting.

Over the last decade, criminal justice scholars have increasingly endorsed "evidence-based practices"; however, some criminologists have voiced concerns over the varied methodological rigor used by evaluation researchers, differing definitions of evidence, and lack of critical exploration as to why programs may be (in)effective. This article argues that evaluability assessments (EAs) can answer these concerns. Through a case study of an EA used on a prison-based fatherhood program, this article demonstrates how EA's approach leads to a more precise understanding of outcome operationalization, and allows for the democratization of research, which is particularly important in a carceral setting.

To evaluate or not: Evaluability study of 40 interventions of Belgian development cooperation.

Due to an increasing importance of evaluations within development cooperation, it has become all the more important to analyse if initial conditions for quality and relevant evaluations are met. This article presents the findings from an evaluability study of 40 interventions of Belgian development cooperation. A study framework was developed focusing on three key dimensions (i.e. theoretical evaluability, practical evaluability and the evaluation context) and subdivided over the different OECD/DAC criteria. Drawing upon a combination of desk and field research, the study framework was subsequently applied on a set of 40 interventions in Benin, DRC, Rwanda and Belgium. Findings highlight that the context dimension scores remarkably better than the theoretical and practical evaluability in particular. The large majority of the interventions have the conditions in place to satisfactorily evaluate effectiveness and efficiency while the opposite holds for sustainability and impact in particular. These findings caution against commissioning of evaluations that ritually focus on all OECD/DAC criteria regardless of their readiness. It underscores the usefulness of a flexible 'portfolio' approach towards evaluations, in which a more systematic use of evaluability assessment from the start of interventions could play a role.

Evaluating Integrative Cancer Clinics With the Claim Assessment Profile: An Example With the InspireHealth Clinic.

BACKGROUND: The evaluation of freestanding integrative cancer clinical programs is challenging and is rarely done. We have developed an approach called the Claim Assessment Profile (CAP) to identify whether evaluation of a practice is justified, feasible, and likely to provide useful information. OBJECTIVES: A CAP was performed in order to (1) clarify the healing claims at InspireHealth, an integrative oncology treatment program, by defining the most important impacts on its clients; (2) gather information about current research capacity at the clinic; and (3) create a program theory and path model for use in prospective research. STUDY DESIGN/METHODS: This case study design incorporates methods from a variety of rapid assessment approaches. Procedures included site visits to observe the program, structured qualitative interviews with 26 providers and staff, surveys to capture descriptive data about the program, and observational data on program implementation. RESULTS: The InspireHealth program is a well-established, multi-site, thriving integrative oncology clinical practice that focuses on patient support, motivation, and health behavior engagement. It delivers patient-centered care via a standardized treatment protocol. There arehigh levels of research interest from staff and resources by which to conduct research. CONCLUSIONS: This analysis provides the primary descriptive and claims clarification of an integrative oncology treatment program, an evaluation readiness report, a detailed logic model explicating program theory, and a clinical outcomes path model for conducting prospective research. Prospective evaluation of this program would be feasible and valuable, adding to our knowledge base of integrative cancer therapies.

Claim Assessment Profile: A Method for Capturing Healthcare Evidence in the Scientific Evaluation and Review of Claims in Health Care (SEaRCH).

BACKGROUND: Grounding health claims in an evidence base is essential for determining safety and effectiveness. However, it is not appropriate to evaluate all healthcare claims with the same methods. "Gold standard" randomized controlled trials may skip over important qualitative and observational data about use, benefits, side effects, and preferences, issues especially salient in research on complementary and integrative health (CIH) practices. This gap has prompted a move toward studying treatments in their naturalistic settings. In the 1990s, a program initiated under the National Institutes of Health was designed to provide an outreach to CIH practices for assessing the feasibility of conducting retrospective or prospective evaluations. The Claim Assessment Profile further develops this approach, within the framework of Samueli Institute's Scientific Evaluation and Review of Claims in Health Care (SEaRCH) method. METHODS/DESIGN: The goals of a Claim Assessment Profile are to clarify the elements that constitute a practice, define key outcomes, and create an explanatory model of these impacts. The main objective is to determine readiness and capacity of a practice to engage in evaluation of effectiveness. This approach is informed by a variety of rapid assessment and stakeholder-driven methods. Site visits, structured qualitative interviews, surveys, and observational data on implementation provide descriptive data about the practice. Logic modeling defines inputs, processes, and outcome variables; Path modeling defines an analytic map to explore. DISCUSSION: The Claim Assessment Profile is a rapid assessment of the evaluability of a healthcare practice. The method was developed for use on CIH practices but has also been applied in resilience research and may be applied beyond the healthcare sector. Findings are meant to provide sufficient data to improve decision-making for stakeholders. This method provides an important first step for moving existing promising yet untested practices into comprehensive evaluation.

Improving the health and safety of 911 emergency call centre agents: an evaluability assessment of a knowledge transfer strategy.

A knowledge transfer (KT) strategy was implemented by the IRSST, an occupational health and safety research institute established in Québec (Canada), to improve the prevention of psychological and musculoskeletal problems among 911 emergency call centre agents. An evaluability assessment was conducted in which each aspect of the KT approach was documented systematically to determine whether the strategy had the potential to be evaluated in terms of its impact on the targeted population. A review of the literature on KT in occupational health and safety and on the evaluation of such KT programmes, along with the development of a logic model based on documentary analysis and semi-structured interviews with key stakeholders, indicated that the KT strategy was likely to have had a positive impact in the 911 emergency call centre sector. Implications for future research are discussed.

Roles of the state asthma program in implementing multicomponent, school-based asthma interventions.

BACKGROUND: Asthma is a leading chronic childhood disease in the United States and a major contributor to school absenteeism. Evidence suggests that multicomponent, school-based asthma interventions are a strategic way to address asthma among school-aged children. The Centers for Disease Control and Prevention (CDC) encourages the 36 health departments (34 states, District of Columbia, and Puerto Rico) in the National Asthma Control Program (NACP) to implement multicomponent, school-based asthma interventions on a larger scale. METHODS: To gain a better understanding of replicable best practices for state-coordinated asthma interventions in schools, an NACP evaluation team conducted evaluability assessments of promising interventions run by state asthma programs in Louisiana, Indiana, and Utah. RESULTS: The team found that state asthma programs play a critical role in implementing school-based asthma interventions due to their ability to (1) use statewide surveillance data to identify asthma trends and address disparities; (2) facilitate connections between schools, school systems, and school-related community stakeholders; (3) form state-level connections; (4) translate policies into action; (5) provide resources and public health practice information to schools and school systems; (6) monitor and evaluate implementation. CONCLUSIONS: This article presents evaluability assessment findings and illustrates state roles using examples from the 3 participating state asthma programs.

Logic models from an evaluability assessment of pharmaceutical services for people living with HIV/AIDS / Modelos lógicos provenientes de estudo de avaliabilidade da assistência farmacêutica para pessoas vivendo com HIV/Aids

Brazil was the first developing country to provide people living with HIV/AIDS (PLWA) with comprehensive, universal, free access to antiretroviral medicines (ARV). Pharmaceutical services are considered a strategic action that has the goal of providing access to rational use of quality medicines while also promoting user satisfaction. User satisfaction is a complex concept, and evaluation models for pharmaceutical services for PLWA were not found in the literature. Therefore, an evaluation approach to help assess this issue had to be developed. This article seeks to describe a theoretical evaluation model of user satisfaction with the dispensing of ARV, developed as part of an Evaluability Assessment (EA). It presents a brief review of the EA and user satisfaction and describes the development of models created during the EA. The lessons learned in the process are presented as a conclusion.

O Brasil foi o primeiro país em desenvolvimento a fornecer medicamentos antiretrovirais (ARV) de forma integral, universal e gratuita às pessoas vivendo com HIV/Aids (PVHA). A Assistência Farmacêutica é considerada uma ação estratégica e busca prover acesso a medicamentos de qualidade, com uso racional, promovendo a satisfação dos usuários. Satisfação do usuá-rio é um conceito complexo e modelos para avaliarem serviços farmacêuticos para PVHA não são encontrados na literatura. Este artigo objetiva descrever o desenvolvimento de três modelos criados em um Estudo de Avaliabilidade (EA). É apresentada uma breve revisão dos conceitos de EA e de Satisfação do Usuário. As lições apreendidas no processo são apresentadas como conclusão.