Getting Rigor Right: A Framework for Methodological Choice in Adaptive Monitoring and Evaluation.

The field of global development has embraced the idea that programs require agile, adaptive approaches to monitoring, evaluation, and learning. But considerable debate still exists around which methods are most appropriate for adaptive learning. Researchers have a range of proven and novel tools to promote a culture of adaptation and learning. These tools include lean testing, rapid prototyping, formative research, and structured experimentation, all of which can be utilized to generate responsive feedback (RF) to improve social change programs. With such an extensive toolkit, how should one decide which methods to employ? In our experience, the level of rigor used should be responsive to the team's level of certainty about the program design being investigated-how certain-or confident-are we that a program design will produce its intended results? With less certainty, less rigor is needed; with more certainty, more rigor is needed. In this article, we present a framework for getting rigor right and illustrate its use in 3 case studies. For each example, we describe the feedback methods used and why, how the approach was implemented (including how we conducted cocreation and ensured buy-in), and the results of each engagement. We conclude with lessons learned from these examples and how to use the right kind of RF mechanism to improve social change programs.

Responsive feedback: Towards a new paradigm to enhance intervention effectiveness.

The current dominant models of intervention design in the development sector do not account for the complexity and unpredictability of implementation challenges. Decision makers and implementers need timely feedback during implementation to respond to field realities and to course-correct. This letter calls for a new approach of "responsive feedback" or "feedback loops" that promotes interactions between project designers, implementers, researchers and decision-makers to enable course corrections needed to achieve intended outcomes. A responsive feedback approach, in theory, should be agile, flexible, adaptive, iterative, and actionable. There can be multiple challenges associated with incorporating this approach into practice including donor requirements, organizational structure and culture, concerns about the additional time required to adopt such an approach, resource and operational constraints, the absence of skill sets needed for such an approach within smaller organizations and inadequate inter-departmental communication. However, these barriers to adaptation can be overcome. For responsive feedback to become a part of the culture of development organizations, commitment is needed from donors, decision-makers, project designers and implementers. We believe that, to generate opportunities for learning and adaptation, donors should provide the stimulus to break down silos between implementers and researchers.

Criteria to assess potential reverse innovations: opportunities for shared learning between high- and low-income countries.

BACKGROUND: Low- and middle-income countries (LMICs) are developing novel approaches to healthcare that may be relevant to high-income countries (HICs). These include products, services, organizational processes, or policies that improve access, cost, or efficiency of healthcare. However, given the challenge of replication, it is difficult to identify innovations that could be successfully adapted to high-income settings. We present a set of criteria for evaluating the potential impact of LMIC innovations in HIC settings. METHODS: An initial framework was drafted based on a literature review, and revised iteratively by applying it to LMIC examples from the Center for Health Market Innovations (CHMI) program database. The resulting criteria were then reviewed using a modified Delphi process by the Reverse Innovation Working Group, consisting of 31 experts in medicine, engineering, management and political science, as well as representatives from industry and government, all with an expressed interest in reverse innovation. RESULTS: The resulting 8 criteria are divided into two steps with a simple scoring system. First, innovations are assessed according to their success within the LMIC context according to metrics of improving accessibility, cost-effectiveness, scalability, and overall effectiveness. Next, they are scored for their potential for spread to HICs, according to their ability to address an HIC healthcare challenge, compatibility with infrastructure and regulatory requirements, degree of novelty, and degree of current collaboration with HICs. We use examples to illustrate where programs which appear initially promising may be unlikely to succeed in a HIC setting due to feasibility concerns. CONCLUSIONS: This study presents a framework for identifying reverse innovations that may be useful to policymakers and funding agencies interested in identifying novel approaches to addressing cost and access to care in HICs. We solicited expert feedback and consensus on an empirically-derived set of criteria to create a practical tool for funders that can be used directly and tested prospectively using current databases of LMIC programs.

Investing in nursing and midwifery enterprise: Empowering women and strengthening health systems--A landscaping study of innovations in low- and middle-income countries.

Women's empowerment and global health promotion are both central aims in the development agenda, with positive associations and feedback loops between empowerment and health outcomes. To date, most of the work exploring connections between health and empowerment has focused on women as health consumers. This article summarizes a much longer landscape review that examines ways in which various health programs can empower women as providers, specifically nurses and midwives. We conducted a scan of the Center for Health Market Innovations database to identify how innovative health programs can create empowerment opportunities for nurses and midwives. We reviewed 94 programs, exploring nurses' and midwives' roles and inputs that contribute to their empowerment. There were four salient models: provider training, information and communications technologies, cooperatives, and clinical franchises. By documenting these approaches and their hallmarks for empowering female health workers, we hope to stimulate greater uptake of health innovations coupled with gender-empowerment opportunities globally. The full report with expanded methodology and findings is available online.

E-health in low- and middle-income countries: findings from the Center for Health Market Innovations.

OBJECTIVE: To describe how information communication technology (ICT) is being used by programmes that seek to improve private sector health financing and delivery in low- and middle-income countries, including the main uses of the technology and the types of technologies being used. METHODS: In-country partners in 16 countries directly searched systematically for innovative health programmes and compiled profiles in the Center for Health Market Innovations' database. These data were supplemented through literature reviews and with self-reported data supplied by the programmes themselves. FINDINGS: In many low- and middle-income countries, ICT is being increasingly employed for different purposes in various health-related areas. Of ICT-enabled health programmes, 42% use it to extend geographic access to health care, 38% to improve data management and 31% to facilitate communication between patients and physicians outside the physician's office. Other purposes include improving diagnosis and treatment (17%), mitigating fraud and abuse (8%) and streamlining financial transactions (4%). The most common devices used in technology-enabled programmes are phones and computers; 71% and 39% of programmes use them, respectively, and the most common applications are voice (34%), software (32%) and text messages (31%). Donors are the primary funders of 47% of ICT-based health programmes. CONCLUSION: Various types of ICT are being employed by private organizations to address key health system challenges. For successful implementation, however, more sustainable sources of funding, greater support for the adoption of new technologies and better ways of evaluating impact are required.

The economics of eHealth and mHealth.

While mHealth has the potential to overcome traditional obstacles to the delivery of health services to the poor in lower and middle-income countries--issues related to access, quality, time, and resources--there is little evidence as to whether the expected benefits and savings can be actualized on a large scale. As a first step to developing the investment case for mHealth, this article outlines some of the key economic and financial questions that need to be answered in developing in-country eHealth investments. The proposed questions focus on the costs of eHealth infrastructure; regulatory structures that provide incentives at different levels of the health delivery system to encourage investment in, and use of, eHealth; and measuring the outcomes of successful eHealth utilization, including anticipated return on investment.