Measurement for Change: Reflections from innovators' experiences with monitoring, evaluation, and learning systems for Early Childhood Development.

In this review paper, we explore how on-the-ground Early Childhood Development (ECD) innovators are using monitoring, evaluation, and learning (MEL) systems to guide the design and implementation of ECD programs, as well as how MEL systems can influence policy and support the achievement of impact at scale. We reflect on articles in the Frontiers series "Effective delivery of integrated interventions in early childhood: innovations in evidence use, monitoring, evaluation, and learning." The 31 contributions to the series reflect the breadth and depth of complexity that characterizes ECD, including global geographic spread, with studies from Asia, Europe, Africa, and Latin America and the Caribbean. Our synthesis finds that integrating MEL processes and systems into the fabric of a program or policy initiative can broaden the underlying value proposition. Specifically, ECD organizations sought to design their MEL systems to ensure programs fit the values, goals, experiences and conceptual frameworks of diverse stakeholders, so that participating makes sense to all. For example, formative, exploratory research identified the priorities and needs of the target population and frontline service providers, and informed the content and delivery of an intervention. ECD organizations also designed their MEL systems to support a shift of accountability toward broader ownership: They included delivery agents and program participants alike as subjects rather than objects, through active participation in data collection, and by providing opportunities for equitable discussion of results and decision-making. Programs collected data to respond to specialized characteristics, priorities and needs, embedding program activities into existing day-to-day routines. Further, papers pointed to the importance of intentionally involving a variety of stakeholders in national and international dialogues to ensure that diverse ECD data collection efforts are aligned and multiple perspectives are considered in the development of national ECD policies. And, several papers illustrate the value of creative methods and measurement tools to integrate MEL into a program or policy initiative. Finally, our synthesis concludes that these findings align with the five aspirations that were formulated as part of the Measurement for Change dialogue, which motivated the launch of the series.

Rapid-Cycle Evaluation in an Early Intervention Program for Children With Developmental Disabilities in South India: Optimizing Service Providers' Quality of Work-Life, Family Program Engagement, and School Enrollment.

Background: This paper explores how implementation and refinement of an early intervention (EI) program for children with delayed development was informed by an iterative, intentional and structured process of measurement. Providing access to early intervention therapy for children in rural areas of India is challenging due to a lack of rehabilitation therapists and programs. Following a biopsychosocial framework and principles of community-based rehabilitation, a non-governmental organization, Amar Seva Sangam (ASSA), overcame those barriers by designing a digital technology supported EI program in rural Tamil Nadu, India. Program objectives included providing service access; supporting program engagement, child development and school enrollment; and positioning the intervention for scale-up. This paper contributes to a growing body of literature on how program design and implementation can be informed through a cyclical process of data collection, analysis, reflection, and adaptation. Methods: Through several strands of data collection, the design and implementation of the EI program was adapted and improved. This included qualitative data from focus groups and interviews with caregivers and service providers, and a mobile application that collected and monitored longitudinal quantitative data, including program engagement rates, developmental progression, caregiver outcomes, and school enrollment status. Results: Measurements throughout the program informed decision-making by identifying facilitators and barriers to service providers' quality of work-life, family program engagement, and school enrollment. Consultation with key stakeholders, including caregivers and service providers, and data driven decision making led to continual program changes that improved service provider quality of work-life, program engagement and school enrollment. These changes included addressing gender-related work challenges for service providers; forming caregiver support networks; introducing psychological counseling for caregivers; providing medical consultations and assistive devices; creating community awareness programs; improving access to therapy services; focusing on caregiver education, motivation and support; and advocacy for accessibility in schools. Conclusion: The process of using evidence-informed and stakeholder driven adaptations to the early intervention program, led to improved service provider quality of work-life, greater program engagement, improved school enrollment and positioned the intervention for scale-up, providing lessons that may be beneficial in other contexts.

The Aspirations of Measurement for Change.

This Perspective presents the five key aspirations of an approach to data use, decision making and monitoring, evaluation, and learning (MEL) in Early Childhood Development (ECD) referred to as Measurement for Change. The core ideas of Measurement for Change gave rise to this series, and many of the papers submitted in this series speak to this approach, whether directly or indirectly. The five aspirations describe interconnected concepts that advocate for practitioners and researchers within ECD to build the capacity to use data in their decision making, by establishing a monitoring, evaluation, and learning system that strives to be: Dynamic; Inclusive; Informative; Interactive; and People-centered.

Measurement for Change: From Idea to Approach.

Measurement for Change proposes an integration of monitoring, evaluation, and learning into decision-making systems that support sustainable transition of interventions to scale. It was developed using a cyclical, interactive 1-year dialogue between early childhood development (ECD) practitioners and academics from across the globe. Details are presented in Krapels et al. (1) as part of this special issue in Frontiers. In this paper, we trace the developments that inspired Measurement for Change and the novel ways in which the approach and the special issue was developed. The experience, and the reflections on this experience, are intended to inform those implementing initiatives that similarly seek to integrate practitioner- and academic experiences in support of sustainable transitions of interventions to scale.

100 Metrics to Assess and Communicate the Value of Biomedical Research: An Ideas Book.

Biomedical research affects society in many ways. It has been shown to improve health, create jobs, add to our knowledge, and foster new collaborations. Despite the complexity of modern research, many of the metrics used to evaluate the impacts of research still focus on the traditional, often academic, part of the research pathway, covering areas such as the amount of grant funding received and the number of peer-reviewed publications. In response to increasing expectations of accountability and transparency, the Association of American Medical Colleges (AAMC), in collaboration with RAND Europe, undertook a project to help communicate the wider value of biomedical research. The initiative developed resources to support academic medical centers in evaluating the outcomes and impacts of their research using approaches relevant to various stakeholders, including patients, providers, administrators, and legislators. This study presents 100 ideas for metrics that can be used to assess and communicate the value of biomedical research. The list is not comprehensive, and the metrics are not fully developed, but they should serve to stimulate and broaden thinking about how academic medical centers can communicate the value of their research to a broad range of stakeholders.

Assessing and Communicating the Value of Biomedical Research: Results From a Pilot Study.

PURPOSE: Assessing the impact of research requires an approach that is sensitive both to the context of the research and the perspective of the stakeholders trying to understand its benefits. Here, the authors report on a pilot that applied such an approach to research conducted at the Collaborative Center for Health Equity (CCHE) of the University of Wisconsin School of Medicine and Public Health. METHOD: The pilot assessed the academic impact of CCHE's work; the networks between CCHE and community partners; and the reach of CCHE's programs, including an attempt to estimate return on investment (ROI). Data included bibliometrics, findings from a stakeholder survey and in-depth interviews, and financial figures. RESULTS: The pilot illustrated how CCHE programs increase the capacity of community partners to advocate for their communities and engage with researchers to ensure that research benefits the community. The results illustrate the reach of CCHE's programs into the community. The authors produced an estimate of the ROI for one CCHE program targeting childhood obesity, and values ranged from negative to positive. CONCLUSIONS: The authors experienced challenges using novel assessment techniques at a small scale including the lack of comparator groups and the scarcity of cost data for estimating ROI. This pilot demonstrated the value of research from a variety of perspectives-from academic to community. It illustrates how metrics beyond grant income and publications can capture the outputs of an academic health center in a way that may better align with the aims of the center and stakeholders.

Mapping the Global Mental Health Research Funding System.

This study maps the global funding of mental health research between 2009 and 2014. It builds from the bottom up a picture of who the major funders are, what kinds of research they support and how their strategies relate to one another. It uses the funding acknowledgements on journal papers as a starting point for this. The study also looks to the future, considering some of the areas of focus, challenges and opportunities which may shape the field in the coming few years.

The future of anticoagulation management in atrial fibrillation in Europe: An assessment of today's challenges with recommendations for the future.

Atrial fibrillation (AF) is the most common type of cardiac arrhythmia, affecting approximately 1-2 per cent of the population worldwide. Those who suffer from AF have a five times higher risk of stroke. AF prevalence increases with age and it affects roughly 18 per cent of the population over 85. Consequently, as populations age, AF is becoming an increasingly significant public health issue. Over recent years there have been developments in treatment and management options, both for treating the arrhythmia directly, and assessing and reducing the risk of AF-related stroke, but there is a need to ensure that available knowledge is applied optimally to benefit patients so that opportunities to prevent AF-related stroke are not missed. The aims of this project were to assess the current landscape and explore the direction of future developments in AF management in Europe, with a focus on the use of anticoagulants in the prevention of AF-related stroke. Through rapid evidence assessment, key informant interviews, PESTLE analysis and the development and exploration of future scenarios, we have developed sets of shorter- and longer-term recommendations for improving AF-related patient outcomes. The short-term recommendations are: i) improve AF awareness among the public and policymakers; ii) support education about AF management for healthcare professionals and patients; and iii) maintain engagement in AF-related research across the health services.

The NIHR Invention for Innovation (i4i) Programme: A Review of Progress and Contributions to Innovation in Healthcare Technologies.

The National Institute for Health Research (NIHR) Invention for Innovation (i4i) programme supports the development of innovative medical technologies for patient benefit. The i4i product development stream involves collaborative projects between at least two partners from academia, the NHS and industry. Medical technology innovators apply for funding for one to three years, through a peer review-based process that includes presentation to a selection panel. The funding and business advice provided by i4i support the development of early-stage innovations, generally at proof of concept and prototype stages. Since its inception the product development stream has identified and supported 170 projects, led by 146 principal investigators (PIs). RAND Europe evaluated the programme, with the aim of identifying its outputs and impacts and examining the factors influencing performance. The evaluation findings should help inform the future of the programme. The evaluation used a multi-method approach, including a focused review of background information from i4i, scoping interviews with key informants, a survey of programme participants and case studies of projects representing diverse technologies and health needs.