RESUMEN
BACKGROUND: This paper reports on an online platform, People's Open Access Education Initiative (Peoples-uni), as a means of enhancing access to master's level public health education for health professionals. Peoples-uni seeks to improve population health in low- and middle-income countries by building public health capacity through e-learning at very low cost. We report here an evaluation of the Peoples-uni programme, conducted within the context of Sustainable Development Goal 4, which seeks to "ensure inclusive and quality education for all and promote lifelong learning" by 2030. The evaluation seeks to address the following three questions: (1) Did Peoples-uni meet its intended goals? (2) What were the different types of impacts that students experienced? (3) What suggestions for future changes in Peoples-uni did students recommend? METHODS: A mixed methods evaluation consisted of two parts, namely an online survey and a telephone interview. A total of 119 master's level graduates were invited to participate; responses were obtained from 71 of those invited, giving a response rate of 60%. Respondents were spread across 31 countries. Interviews were conducted with 18 respondents. RESULTS: There was strong evidence that Peoples-uni had achieved its stated goals. Potential impacts on students included knowledge to enhance practice and appreciation of context, enhanced research capacity through knowledge of public health, critical thinking and evidence-based programming, and empowerment of students about the potential of education as a means of improving their lives. Accreditation through future partnerships with local universities was recommended by students. CONCLUSIONS: Peoples-uni has been able to deliver a credible public health master's level educational programme, with positive impacts on the students who graduated. Challenges are to find a way to accredit the programme to ensure its sustainability and to see how to take full advantage of the current, and future, graduates to turn this from an education programme into a capacity-building programme with real impact.
Asunto(s)
Educación de Postgrado/métodos , Educación Profesional/métodos , Objetivos , Internet , Evaluación de Programas y Proyectos de Salud , Salud Pública/educación , Estudiantes , Acreditación , Creación de Capacidad , Curriculum , Práctica Clínica Basada en la Evidencia , Personal de Salud , Humanos , Aprendizaje , Investigación/educación , Encuestas y Cuestionarios , Desarrollo Sostenible , UniversidadesRESUMEN
The papers in this volume grapple with various issues related to the use of theories of change in program evaluation. This introductory paper reviews some of the critical challenges that arise in developing and learning from theory-driven evaluations. These challenges include the relationship between theories of change and ecologies of evidence, the need for epistemic fluency in learning, and coming to terms with the initial incompleteness of knowledge in program mechanisms. The nine papers that follow, which represent a geographically diverse set of evaluations and evaluators (including Scotland, India, Canada, USA), help develop these and other themes. This volume of papers also serves as a celebration of the work of John Mayne, one of the foremost theory-driven evaluators of the last few decades. John passed away in December of 2020. This volume is intended to honor his legacy while also identifying challenging issues that call for further development.
Asunto(s)
Conocimiento , Humanos , Evaluación de Programas y Proyectos de Salud , Canadá , IndiaRESUMEN
This paper highlights how learnings from exploring assumptions can be strengthened by taking an evolutionary approach to theory building and analysis. We discuss theory-driven evaluation applied to a community-based intervention implemented by Dancing With Parkinson's in Toronto, Canada, targeting Parkinson's disease (PD), a neurodegenerative condition affecting movement. A major gap in the literature is understanding the mechanisms by which dance might make a difference in the daily lives of people living with PD. This study was an early exploratory evaluation to better understand mechanisms and short-term outcomes. Conventional thinking generally favors "permanent" over "transitory" changes, and "long-term" over "short-term" effects. Yet, for people living with degenerative conditions (and also people experiencing chronic pain and other chronic symptoms), transitory and short-term changes may be highly valued and welcomed relief. In order to study and link multiple longitudinal events to explore key linkages in the theory of change, we piloted the use of diaries, with brief entries filled out daily by participants. The aim was to better understand the short-term experiences of participants using their daily routines as a means of learning about potential mechanisms, what matters to participants, and to see if small effects could be observed on days when participants danced versus days when they did not dance and also longitudinally over several months. Our initial theoretical stance began with a view of dance as exercise and the well-established benefits of exercise; yet, we explored through the diary data collected, as well as client interviews and literature review, potential other mechanisms of dancing (such as group interaction, touch, stimulation by the music, and esthetics including "feeling lovely"). This paper does not develop a full, comprehensive theory of dance but moves towards a more comprehensive view that locates dance within the routine activities of participants' daily lives. We argue that given the challenges of evaluating complex interventions comprising multiple, interacting components, there is a need for an evolutionary learning process to understand heterogeneities in mechanisms -- what works for whom -- when faced with knowledge incompleteness in the theory of change.
Asunto(s)
Danzaterapia , Enfermedad de Parkinson , Humanos , Evaluación de Programas y Proyectos de Salud , Ejercicio Físico/fisiología , EmocionesRESUMEN
This paper discusses the central importance of involving program recipients in the initial development of the theory of change. Through an example of a drop-in located in the inner-city of Toronto, we describe how, first, theories of change based on funder and even staff perspectives may have program goals that do not connect with the lives and values of clients. Second, engagement with clients should surface heterogeneities in their expectations; many theories of change, even when they are developed with clients, assume there is a 'homogenous' clientele. Third, programs that address the needs of marginalized individuals need to pay attention to the dynamics of marginalization. These points have consequences for the development of the initial theories of change.
Asunto(s)
Terapia Conductista , Objetivos , Evaluación de Programas y Proyectos de Salud , HumanosRESUMEN
Background: The Lancet Global Health Commission (LGHC) has argued that quality of care (QoC) is an emergent property that requires an iterative process to learn and implement. Such iterations are required given that health systems are complex adaptive systems.Objective: This paper explores the multiple roles that evaluations need to play in order to help with iterative learning and implementation. We argue evaluation needs to shift from a summative focus toward an approach that promotes learning in complex systems. A framework is presented to help guide the iterative learning, and includes the dimensions of clinical care, person-centered care, continuum of care, and 'more than medicine. Multiple roles of evaluation corresponding to each of the dimensions are discussed.Methods: This paper is informed by reviews of the literature on QoC and the roles of evaluation in complex systems. The proposed framework synthesizes the multiple views of QoC. The recommendations of the roles of evaluation are informed both by review and experience in evaluating multiple QoC initiatives.Results: The specific roles of different evaluation approaches, including summative, realist, developmental, and participatory, are identified in relationship to the dimensions in our proposed framework. In order to achieve the potential of LGHC, there is a need to discuss how different evaluation approaches can be combined in a coherent way to promote iterative learning and implementation of QoC initiatives.Conclusion: One of the implications of the QoC framework discussed in the paper is that time needs to be spent upfront in recognizing areas in which knowledge of a specific intervention is not complete at the outset. This, of course, implies taking stock of areas of incompleteness in knowledge of context, theory of change, support structures needed in order for the program to succeed in specific settings. The role of evaluation should not be limited to only providing an external assessment, but an important goal in building evaluation capacity should be to promote adaptive management among planners and practitioners. Such iterative learning and adaptive management are needed to achieve the goals of sustainable development goals.
Asunto(s)
Aprendizaje , Calidad de la Atención de Salud , Salud Global , Humanos , Desarrollo SostenibleRESUMEN
This paper describes some of the main challenges of evaluating complex interventions, as well as the implications of such challenges for evaluation capacity building. It discusses lessons learned from a case study of an evaluation of Dancing with Parkinson's, an organization that provides dance classes to people with Parkinson's disease in Toronto, Canada. These implications are developed from a realist evaluation lens. Key lessons include the need to develop skills to understand program mechanisms and contexts, recognize multiple models of causality, apply mixed method designs, and ensure the successful scaling up and spread of an intervention.
Asunto(s)
Creación de Capacidad , Enfermedad de Parkinson , Canadá , Humanos , Evaluación de Programas y Proyectos de Salud , Proyectos de InvestigaciónRESUMEN
This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson's. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).
Asunto(s)
Creación de Capacidad , Enfermedad de Parkinson , Humanos , Organizaciones , Enfermedad de Parkinson/diagnóstico , Enfermedad de Parkinson/terapia , Aprendizaje Basado en Problemas , Evaluación de Programas y Proyectos de SaludRESUMEN
BACKGROUND: This paper explores the heterogeneities in antenatal care (ANC) utilization in India's most populated state, Uttar Pradesh. Taking an intersectionality lens, multiple individual- and district-level factors are used to identify segments of any antenatal care usage in Uttar Pradesh Objective: This paper seeks to understand the multilevel contexts of ANC utilization. The planning and programming challenge is that such knowledge of contextual specificity is rarely known upfront at the initial stages of planning or implementing an intervention. Exploratory data analysis might be needed to identify such contextual specificity. METHODS: Tree-structured regression methods are used to identify segments and interactions between factors. The results from the tree-structured regression were complemented with multilevel models that controlled for the clustering of individuals within districts. RESULTS: Heterogeneities in utilization of any ANC were observed. The multiple segments of ANC utilization that were developed went from a low utilization of 23.7% for respondents who were not literate and did not have home ownership to a high of 82.4% for respondents who were literate and at the highest level of wealth. Key variables that helped define the segments of ANC utilization include: woman's literacy, ownership of home, wealth index, and district-level sex ratio. Based on the multilevel model of any ANC utilization, cross-level interactions also were obtained between sex ratio and ownership of home as well as between sex ratio and literacy. Increases in sex ratio increased the influence of ownership of home on any ANC, while increases in sex ratio reduced the impact of woman's literacy on receiving any ANC. CONCLUSION: We argue that a focus on heterogeneous segments of utilization can help build knowledge of the mechanisms that underlie inequities in maternal health utilization. Such knowledge of heterogeneity needs to be incorporated in contextualizing interventions to meet a variety of recipients' needs.
Asunto(s)
Atención Prenatal/estadística & datos numéricos , Adulto , Femenino , Humanos , India , Alfabetización , Salud Materna , Análisis Multinivel , Embarazo , Análisis de Regresión , Factores SocioeconómicosRESUMEN
This paper describes a framework that can help refine program theory through data explorations and stakeholder dialogue. The framework incorporates the following steps: a recognition that program implementation might need to be multi-phased for a number of interventions, the need to take stock of program theory, the application of pattern recognition methods to help identify heterogeneous program mechanisms, and stakeholder dialogue to refine the program. As part of the data exploration, a method known as developmental trajectories is implemented to learn about heterogeneous trajectories of outcomes in longitudinal evaluations. This method identifies trajectory clusters and also can estimate different treatment impacts for the various groups. The framework is highlighted with data collected in an evaluation of an alcohol risk-reduction program delivered in a college fraternity setting. The framework discussed in the paper is informed by a realist focus on "what works for whom under what contexts." The utility of the framework in contributing to a dialogue on heterogeneous mechanism and subsequent implementation is described. The connection of the ideas in paper to a 'learning through principled discovery' approach is also described.
Asunto(s)
Alcoholismo/prevención & control , Promoción de la Salud/organización & administración , Modelos Teóricos , Desarrollo de Programa/métodos , Femenino , Humanos , Masculino , Estudios de Casos Organizacionales , Evaluación de Programas y Proyectos de Salud/métodos , Conducta de Reducción del Riesgo , Investigación Biomédica Traslacional , Universidades , Adulto JovenRESUMEN
BACKGROUND: Mainstreaming of gender, equity, and human rights (GER) is an important focus of the World Health Organization (WHO) and other UN organizations. This paper explores the role of action plans in mainstreaming GER. This paper is informed by a theory-driven evaluation lens. DESIGN: A theory of change framework explored the following seven dimensions of how action plans can implement mainstreaming of GER: awareness of the foundations of GER; understanding of context; planning to impact GER; implementation for GER; monitoring, evaluation, and learning; planning for sustainability; agenda setting and buy-in. The seven dimensions were used to analyze the action plans. Reviewers also explored innovations within each of the action plans for the seven dimensions. RESULTS: GER mainstreaming is more prominent in the foundation, background, and planning components of the plan but becomes less so along the theory of change including implementation; monitoring and evaluation; sustainability; and agenda setting and buy-in. CONCLUSIONS: Our analysis demonstrates that much more can be done to incorporate GER considerations into the action planning process. Nine specific recommendations are identified for WHO and other organizations. A theory-driven approach as described in the paper is potentially helpful for developing clarity by which action plans can help with mainstreaming GER considerations.
RESUMEN
While there is a growing literature on building performance measurement systems for health equities, this literature for the most part has not dealt with the challenges of coordinating the various parts of the system, the heterogeneous nature of such systems, or how evaluations and measurement can themselves improve performance. This paper describes the initial steps taken to build a performance measurement system to coordinate health equity across 18 hospitals led by the Toronto Central Local Health Integration Network, which is a regional health authority serving a population of more than 2.5 million residents (near in population to Chicago and Rome) and the most socially diverse urban network in Ontario, Canada. This paper also describes some principles that can help inform a performance measurement system. The innovative aspect of this paper is that these principles were developed through feedback by the hospitals.
Asunto(s)
Atención a la Salud/organización & administración , Investigación sobre Servicios de Salud/organización & administración , Disparidades en el Estado de Salud , Evaluación de Procesos y Resultados en Atención de Salud/organización & administración , Canadá , Humanos , Indicadores de Calidad de la Atención de SaludRESUMEN
This paper proposes ten steps to make evaluations matter. The ten steps are a combination of the usual recommended practice such as developing program theory and implementing rigorous evaluation designs with a stronger focus on more unconventional steps including developing learning frameworks, exploring pathways of evaluation influence, and assessing spread and sustainability. Consideration of these steps can lead to a focused dialogue between program planners and evaluators and can result in more rigorously planned programs. The ten steps can also help in developing and implementing evaluation designs that have greater potential for policy and programmatic influence. The paper argues that there is a need to go beyond a formulaic approach to program evaluation design that often does not address the complexity of the programs. The complexity of the program will need to inform the design of the evaluation. The ten steps that are described in this paper are heavily informed by a Realist approach to evaluation. The Realist approach attempts to understand what is it about a program that makes it work.