The GiveWell Blog

A major update in our assessment of water quality interventions

As we continue to grow, GiveWell seeks to maximize both the cost-effectiveness of the funding we direct and the likely room for more funding of the programs we support. We think we’ve identified a category of interventions that rates really well on both: water treatment, such as chlorination.

This is a major update for us. Before 2020, based on the available evidence, we didn’t believe that water quality interventions had a large enough effect on mortality to make them a competitive target for funding. We’ve since seen new evidence that has led us to significantly increase our estimate of the mortality reduction in young children that’s attributable to these interventions: a 14% reduction in mortality from any cause,[1] up from around 3%.

Though we have remaining uncertainties about these numbers, we’ve substantially updated our view of the promisingness of water treatment. Where we previously found that Evidence Action’s Dispensers for Safe Water program was about as cost-effective as unconditional cash transfers, we now believe it’s about four to eight times as cost-effective, depending on the location. That was a primary factor in our decision to recommend a grant of up to $64.7 million to Dispensers for Safe Water in January 2022.

We’re sharing this news in brief form before we’ve published a grant page, because we’re excited about the potential of this grant and what it represents. It’s an area of work we haven’t supported to a significant degree in the past, but one that we now think could absorb hundreds of millions of dollars in funding for cost-effective programming.

The problem and the intervention

In low-income settings, contaminated water is a major cause of diarrhea, a leading cause of death in children under five years old.[2] Several interventions exist to either purify water or protect it from contamination in the first place, but chlorination has a number of features that made it an attractive intervention for us to explore: it is inexpensive and widely used, several charities are already set up to implement it, and the technology behind it is well established.[3]

Chlorine is well-known as a disinfectant; it reacts with disease-causing microorganisms in water, inactivating viruses and bacteria.[4] There is evidence that chlorination programs, such as distributing chlorine to households, reduce diarrhea in children, but there had been scant evidence that such interventions reduce mortality, which is the single largest driver of cost-effectiveness in our models of water quality interventions.[5] Before 2020, we estimated the effect chlorination had on the prevalence of diarrhea, and then extrapolated from that effect to arrive at an indirect mortality reduction estimate of roughly 3%.[6] This translated to a low cost-effectiveness estimate relative to the programs GiveWell typically funds.

What led us to update

In mid-2020, Michael Kremer, one of the researchers who conducted the ​​initial trial studying dispensers for safe water and now a researcher at the University of Chicago who has extensively studied global health interventions, shared with us preliminary results from his team’s new meta-analysis of mortality data from randomized controlled trials (RCTs) of water quality interventions.[7] (The working paper has since been published, here.) Pooling studies allowed Kremer’s team to obtain statistically significant results from studies that weren’t individually large enough to produce such results on their own. These results suggest that water treatments have a significant effect on mortality from any cause (“all-cause mortality”) in children under five—Kremer et al. currently estimate a roughly 25% reduction.[8]

Their finding is especially noteworthy because it comes from an analysis of direct evidence on mortality, and it is eight times larger than an indirect estimate that extrapolates from diarrhea morbidity. This implies that water treatment yields a mortality benefit that cannot be fully explained by reduction in diarrhea alone.

This led us to take a deeper look at the evidence and revisit our cost-effectiveness analysis. Using the new data as a foundation, we completed our own meta-analysis of RCTs of chlorination interventions, arriving at a mortality reduction estimate in children under five of about 14%.[9] After adjustments, which were informed by (among other things) local diarrhea mortality rates and differences between the interventions used in the trials versus in the charity context, we estimate that two types of chlorination interventions—Dispensers for Safe Water and in-line chlorination (both described below)—reduce all-cause mortality in under-five children by 6 to 11%,[10] depending on the program and location. This increase in mortality reduction, combined with the additional benefits modeled in our current analysis,[11] corresponded with an increase in cost-effectiveness, which made these programs look much more promising.

In a future blog post, we’ll explain more about what led us to complete our own analysis of the mortality data. More on the process we undertook to arrive at our estimate is in our water quality intervention report.

The program

Dispensers for Safe Water installs, maintains, and promotes use of chlorine dispensers at rural or remote water collection sites in Uganda, Malawi, and Kenya. Our 2018 report on Dispensers for Safe Water concluded that, due to our uncertainty over mortality effects at the time and a model that found Dispensers for Safe Water to be about as cost-effective as unconditional cash transfers, it didn’t meet our bar for directing funding.[12]

When we revisited Dispensers for Safe Water after our re-analysis of mortality effects, it began to look much more cost-effective—between four and eight times as cost-effective as unconditional cash transfers, depending on location.[13] That new estimate, combined with our confidence in Evidence Action as an implementer[14] and our understanding from conversations with Evidence Action and others that water treatment is a particularly neglected focus area, led us to recommend a grant to Dispensers for Safe Water of up to $64.7 million[15] over a period of seven years.

Longer-term, we think there’s a true funding gap for this work that’s unlikely to be filled by other funders. We speculatively estimate that Dispensers for Safe Water, or programs targeting similar populations at a comparable level of cost-effectiveness, could cost-effectively use $350 million per year globally.[16] We’ll publish more details in our forthcoming write-up of the grant.

Other water quality interventions

Our current view of water treatment’s effectiveness at reducing mortality and its relative neglectedness has led us to begin exploring water quality interventions besides Dispensers for Safe Water. We’ve reached a fairly advanced stage in our investigation of one such intervention, Evidence Action’s in-line chlorination program, a technology that automatically disinfects the water in communal water tanks with chlorine rather than requiring an individual user to add it.[17] We’re in much earlier stages of researching other programs, such as distributing vouchers for at-home chlorination supplies, distributing water filters, and protecting natural springs from contamination.

In conclusion

We’re excited about the potential of this grant to Dispensers for Safe Water to maintain and expand on a much-needed, life-saving service, and we’re hopeful that water treatment generally will provide even more opportunities for us to do good with our growing resources. Our research is far from done—due to our remaining uncertainties, we want to fund additional studies on the effects of water treatment to help us refine our mortality reduction estimate and pressure-test our current findings. This is an integral part of GiveWell’s commitment to truth-seeking–seeking out new data and being open to revising our views.

For more on our investigation, see our intervention report on water quality. More details on the Dispensers for Safe Water grant will be available in a forthcoming grant page.

Comments

  • Hi GiveWell, thanks for all you’re doing!

    Quick question: The upper bound of your meta-analysis estimate reported in the spreadsheet[1] is actually positive (risk ratio of 1.10), meaning a *detrimental* effect (i.e., a increase in deaths of 10%) from the intervention. This is ignored in your model, since you only use the point estimate (RR of 0.86). Any thoughts on that? Maybe you have good reasons, I’m just curious.

    [1] https://docs.google.com/spreadsheets/d/1dxRfhPnTBJ_UYAeoAyVd8u6Orxh4pR9hB3wvdfZcXFs/edit#gid=255622053

  • Ethan Kennerly on April 9, 2022 at 8:53 pm said:

    The transparent spreadsheet that TB linked reaffirmed my confidence in the meticulous meta-review by GiveWell.

    This cogent blog post on evaluating medicine touched my heart.

  • Miranda Kaplan on April 14, 2022 at 1:21 pm said:

    Hi, TB,

    Thanks for your question! You are correct that the 95% confidence interval we report in our meta-analysis is wide; it allows for as much as a 32% reduction in all-cause mortality (see this cell) or, as you point out, as much as a 10% increase in same. This reflects the statistical uncertainty of the estimate that results from our method of pooling data.

    That said, we think it is very unlikely that chlorination interventions actually lead to an increase in mortality. Chlorination is a well-characterized technology for purifying water, and we don’t know of any ways that chlorination interventions could plausibly lead to an increase in deaths in this context. By contrast, the way chlorination would work to reduce deaths—by reducing exposure to bacteria and viruses in water that cause diarrhea—is well understood. There is also reasonably strong evidence from RCTs that chlorination reduces diarrhea, which is a leading cause of death among young children in low-income settings with unsafe water, so we feel fairly confident that chlorination does reduce deaths in those populations.

    All that to say, our meta-analysis was an attempt to quantify the effect of chlorination interventions on mortality as best we could, but the estimate we arrived at wasn’t the only factor in our assessment—as we generally do, we also qualitatively considered the plausibility of a mortality benefit based on our understanding of how the intervention works (as well as RCT evidence of effects on morbidity). I hope that’s helpful!

  • Ethan Kennerly on June 19, 2022 at 7:52 pm said:

    From my humble resources, I support the optimistic grant. Yet the meta-analysis of Water Quality Interventions thoughtfully acknowledged unquantified uncertainty and other limitations. I imagine GiveWell feels more confident in the evidence of their Top Charities. By 2032, what evidence will GiveWell monitor that might potentially reduce the optimism on how efficiently chlorination averted infant mortality?

  • Miranda Kaplan on June 28, 2022 at 12:02 pm said:

    Hi, Ethan,

    Thanks for your support! Part of the grant we recommended to Dispensers for Safe Water will support rigorous monitoring and evaluation, which we think will provide more precise inputs into our cost-effectiveness estimate. Planned M&E activities include dispenser installation surveys and spot checks of functionality, to inform the total number of working dispensers; high-quality data collection on number of households and people served; and representative sampling to check the chlorination rate of treated water and how many people chlorinate their water at baseline (before dispensers are installed). This data should help us get a better estimate of the total program costs and benefits, and thus help us understand whether the program is more, less, or as cost-effective as we thought when we recommended the grant (and we do expect to start getting this information well before 2032!). We’ll publish more about the monitoring and evaluation activities Evidence Action will carry out on a forthcoming grant page.

    We’re also investigating opportunities to potentially support other research that will help us better understand the link between water quality improvements and mortality. We still have a lot of uncertainty about the mortality effect size estimate we arrived at in our cost-effectiveness analysis, and reduction in mortality is the single largest benefit we model from water quality interventions, so getting an update on this would be very valuable to us.

Comments are closed.