Exogenous factors matter when interpreting the results of an impact evaluation: a case study of rainfall and child health programme intervention in Rwanda.

OBJECTIVE: Public health interventions are often implemented at large scale, and their evaluation seems to be difficult because they are usually multiple and their pathways to effect are complex and subject to modification by contextual factors. We assessed whether controlling for rainfall-related variables altered estimates of the efficacy of a health programme in rural Rwanda and have a quantifiable effect on an intervention evaluation outcomes. METHODS: We conducted a retrospective quasi-experimental study using previously collected cross-sectional data from the 2005 and 2010 Rwanda Demographic and Health Surveys (DHS), 2010 DHS oversampled data, monthly rainfall data collected from meteorological stations over the same period, and modelled output of long-term rainfall averages, soil moisture, and rain water run-off. Difference-in-difference models were used. RESULTS: Rainfall factors confounded the PIH intervention impact evaluation. When we adjusted our estimates of programme effect by controlling for a variety of rainfall variables, several effectiveness estimates changed by 10% or more. The analyses that did not adjust for rainfall-related variables underestimated the intervention effect on the prevalence of ARI by 14.3%, fever by 52.4% and stunting by 10.2%. Conversely, the unadjusted analysis overestimated the intervention's effect on diarrhoea by 56.5% and wasting by 80%. CONCLUSION: Rainfall-related patterns have a quantifiable effect on programme evaluation results and highlighted the importance and complexity of controlling for contextual factors in quasi-experimental design evaluations.