Providing clean water to people living in developing countries is a cause that many donors are interested in. Among other hardships, unclean water can lead to diseases such as diarrhea, which is responsible for millions of child deaths annually. Unfortunately, we have found little evidence that charities’ efforts to improve water infrastructure in the developing world have resulted in decreased incidence of diarrhea and we have not identified a water charity we can confidently recommend to donors.

A recent randomized controlled trial (RCT) Spring cleaning: A Randomized Evaluation of Source Water Quality Improvement, by J-PAL researchers, however, provides the first strong evidence we’ve seen for the impact of a water infrastructure project. Unprotected springs in western Kenya were randomly assigned to receive “protection” at the beginning or the end of the study period. “Protection” involved encasing the spring in concrete and directing water through a pipe so that water did not come into contact with contaminants on the ground. The researchers found that children under the age of 3 living in households that used protected springs were 25% less likely to have had an episode of diarrhea in the last week than children in households that used unprotected springs.

Note that an earlier draft of this study found no statistically significant impact on diarrhea incidence, so the update is a new and somewhat surprising piece of evidence.

So the project appears to have worked; what about the cost-effectiveness? This is tricky to assess:

• Any calculation of the cost-effectiveness of spring protection will be highly dependent on how many households with young children use the spring and how long the spring remains useable.
• The study’s authors estimate that “the cost per DALY averted for this intervention is $16.75.” Using a simple conversion calculation, we estimate that this is equivalent to about $0.25 per case of diarrhea averted and $534 per death adverted, which is well within the range we consider highly cost-effective.
• However, the authors provide no information on how they reached this figure, and our own attempts to reconstruct the calculation (found in this spreadsheet) yield estimates ranging from $0.32 to $1.88 per diarrhea case averted ($21.63-$126.04 per DALY averted; $689.52-$4018.34 per death averted) depending on which assumptions are used. The more reasonable of our estimates (incorporating “discounting” of future diarrhea cases to mirror discounting of costs, and assuming that 75% or fewer of children under 12 are also under 3) imply costs of at least $0.61 per case / $40.66 per DALY / $1,296.24 per death averted – several times more expensive than we estimate it takes to prevent a death from a vaccine-preventable disease (~$200), from tuberculosis (~$150-$750), or from malaria (~$182-$1126). Update: We made an error. The authors do provide information about how they reached their estimate.

  They used a different approach than ours, and having considered it, we defer to their estimate as likely more accurate.

  We believe the source of the difference in our cost-effectiveness estimates is due to a difference in the conversion calculation used to translate between cases, DALYs, and deaths averted. The study’s authors used a conversion factor of 864 cases averted = 32 DALYs averted = 1 death averted, based on data on child deaths from diarrhea in sub-Saharan Africa from from the Global Burden of Disease report (2006, data from 2001) and data on cases of diarrhea per child in sub-Saharan Africa from Kirkwood (1991) (Pg 21). We used a conversion factor of 2136 cases averted = 67 DALYs averted = 1 death averted, based on data on total deaths, DALYs, and cases lost due to diarrhea from the World Health Organization for 2004. As we now believe that the study’s authors made more reasonable assumptions than we did, we defer to their estimate and conclude that, under conditions similar to the ones in the study, spring protection is may be a highly cost-effective way to save lives in sub-Saharan Africa.

Regardless, this study is good news for donors interested in improving health through water infrastructure. Independent evidence of effectiveness lowers the burden of proof somewhat for a charity conducting spring protection. Still, we feel the burden of proof for such a charity remains higher than for one delivering simple, proven medical interventions, such as vaccines or treatments for tuberculosis, malaria, or intestinal worms. Here’s why:

• Medical interventions are generally subjected to extensive, highly rigorous testing before being approved for use. There is currently only one rigorous study (that we know of) of the impact of spring protection.
• We would guess that there is more variation in the protection of springs than in the administration of proven medical interventions. While some medical interventions may be complex, many, such as vaccines and tuberculosis treatment, involve providing a chemical compound in a standardized way. How spring protection is carried out, and therefore how closely it resembles the intervention studied in the RCT, is likely dependent on such variable characteristics as the physical features of the spring, locally available building materials, and the particular design chosen by the charity.
• The effect of spring protection is also likely more variable than the effect of medical interventions. Local practices such as how water is stored, whether water is boiled before use, what percentage of each household’s water comes from the spring, and whether defecation takes place near springs are likely to affect how large of a determinant of diarrhea incidence spring protection is.

Four out of the eleven charities listed at our overview of water programs appear to do at least some spring protection, though none appear to focus on spring protection.* In investigating a charity working on spring protection, we would want to see evidence that the springs are monitored over time to determine whether springs remain in use and in good condition. Ideally, we would also like to see that disease rates have declined among users of protected springs in relation to a suitable comparison group. From what we’ve seen to date, this is a bar that water charities have yet to reach.

*Specifically: