The GiveWell Blog

Reservations about water quality interventions

When we started researching water quality interventions, we originally guessed that there was clear evidence that the programs were effective at improving people’s health based on our general intuitions about the programs and our initial read of the relevant Cochrane review’s bottom line. Once we dug into the details of the research, though, we realized that the evidence regarding the program is mixed — a major reason that we don’t currently recommend charities that work on improving water quality.

This post describes that research, and is drawn from our full report on water quality interventions, which provides more detail and citations. The goal of this blog post is to summarize that report in a more accessible form.

The key points of this post:

  • We formed an initial impression that water quality interventions (e.g., chlorine tablets or dispensers; water filters; other types of treatment that clean drinking water in the developing world) improve people’s health, based on our understanding of the dangers of contaminated water. However, digging into the evidence regarding clean water led us to conclude that the evidence base is mixed.
  • We’re not confident that water quality interventions are not effective; rather, we believe that it’s unclear whether or not they’re effective, and so we don’t include water quality among our priority programs.

The intuitive case for water quality charities

Intuitively, providing a method for cleaning water seems like it should be an effective way to improve health. There is little debate that contaminated water causes diarrhea or that the methods used to treat water reduce contamination, and so it’s reasonable to expect that interventions that reduce water contamination ought to reduce diarrhea. And because diarrhea is deadly – the Global Burden of Disease estimated that in 2010, diarrheal disease caused about 10% of child deaths in developing countries – these interventions could be potentially life-saving.

Developing world randomized controlled trials

A number of independent studies have been done on water quality programs. The results of those studies look very promising: They largely find that providing clean water reduces diarrhea rates.

Specifically, 20 randomized controlled trials conducted in developing countries with around 30,000 participants found that interventions to improve water quality were effective at preventing diarrhea, with an average 35% reduction in the odds of getting diarrhea, according to a meta-analysis (study of studies) conducted by the Cochrane Collaboration. Relative to most interventions we’ve considered, twenty randomized controlled trials is an unusually high number of studies.

We typically consider randomized controlled trials to be a particularly strong method of studying global health outcomes. These studies compare a group that receives an intervention – in this case, some method for cleaning water, like chlorine tablets – with a control group that doesn’t. The intervention and control groups are randomly chosen to try to ensure that two groups are similar in every way other than whether or not they receive the intervention. This way, any differences between the groups ought to be due to the intervention.

The Cochrane Collaboration’s findings, then, strongly suggest that the cleaner water is responsible for the fall in diarrhea rates.

Possible weakness of the randomized controlled trials

When we examined these studies more closely, we noticed a potential weakness. As the Cochrane review states: “Only four of the 22 randomised controlled studies, however, were properly blinded”. (22 because the Cochrane review also considers 2 studies conducted in the United States, which we excluded from our analysis.) In other words, the researchers conducting these studies measured diarrhea rates by asking participants to report them, and the study participants knew whether or not they had received the intervention that was being tested.

It’s possible that this knowledge influenced how participants reported diarrhea cases. For example, participants who knew that their water was being treated may have been less likely to report cases of diarrhea, perhaps because they recalled using the treatment and misremembered how many cases they had, or because they anticipated what the surveyor wanted to hear. If that were the case, the study would find an inaccurately low number of diarrhea cases among people that received the clean water intervention, which would result in the study finding an inflated effect of the intervention.

To deal with the issue of participants knowing whether or not they received the intervention (and the possibility that this influences reporting), studies can use “blinding” to disguise who has received the intervention by distributing a placebo (for example, a fake, non-functional water filter) to the control group. When this method is used, participants don’t know whether they have the fake filter or the real one, and so any bias in how they report their cases of diarrhea ought to affect both groups equally, so we can still measure the difference between the two groups without this source of bias affecting the outcome.

Blinded randomized controlled trials should let us know whether the tested intervention is actually improving people’s health, or whether other sources of harmful pathogens are negating any positive effect of the clean water programs. If the latter were true, we may not expect supporting the intervention to have a beneficial impact on the population.

Blinded studies

We identified five blinded studies of water quality interventions that have been conducted in developing countries (which includes some published after the Cochrane review). These studies measured household interventions to clean water; four studied chlorination and one examined the impact of water filters. They did not find a statistically significant effect of the interventions on diarrhea. (Pooling the effects of the studies, we calculated a 5% reduction in the percent of days with diarrhea for children under 5, with a 95% confidence interval ranging from a 6% increase to a 14% decrease, and no reduction for all ages with a 95% confidence interval ranging from a 12% increase to an 11% decrease.)

This suggests that biased reporting from study participants who knew whether or not they were receiving an intervention may explain why the non-blinded studies in the Cochrane review found a statistically significant effect – that is, it suggests that people in the non-blinded studies may have underreported diarrhea rates when they knew that they were receiving a treatment, but when people were unsure whether or not they were receiving a treatment, there was no reporting bias and so no statistically significant effect was found.

Possible weaknesses of the blinded studies

However, these blinded studies have their own weaknesses. In one, the fake filter used for the control group was actually somewhat effective at cleaning the water, meaning that the control group (which wasn’t supposed to be receiving any clean-water benefits) received water that was at least partially cleaned. It’s not surprising, then, that the study didn’t find a noticeable difference between the diarrhea rates of the two groups.

Two of the other blinded studies were conducted in groups that already had low levels of diarrheal disease, which may have made it harder to detect a difference between the control and intervention groups. The other two had small sample sizes, of 112 (the smallest included in the Cochrane review) and 287 people. (Additional problems with the blinded studies are discussed here.)

These problems suggest that the blinded studies are not conclusive in determining that there’s no statistically significant effect of providing chlorine and water filters to reduce diarrhea rates – each of those problems might explain why the blinded studies found no statistically significant effect, even if the intervention itself is actually beneficial.

(There are a handful of other, different studies that we do not find compelling – you can read more about those here.)

How could these interventions fail to reduce diarrhea rates?

Considering the scientific consensus that contaminated water can cause diarrhea, one could reasonably ask whether it’s even plausible that water quality interventions have small or negligible effects (as opposed to large effects). Is it? There is a potential explanation: Other pathways for pathogens, such as from fecal matter on unwashed hands or from food via flies or other people’s hands, especially in an environment with poor sanitation, may cause a significant number of diarrhea cases, so it’s possible that the gains from simply providing clean water could be small if people are regularly infected via other sources. In fact, it’s possible that water quality interventions might have little effect on diarrhea unless all sources of pathogens are eliminated. The possibility that providing clean water will have little effect in an environment with other sources of pathogens is especially concerning because the blinded studies found no statistically significant effect of the intervention.

Bottom line

We concluded in our intervention report (emphasis included for this post),

The strength of the evidence relies on how one chooses to weigh a few, blinded studies with some methodological weaknesses that report no effect against a large number of unblinded studies, some of which may also have some methodological weaknesses, that, on average, report large effects… Overall, we are ambivalent about the effect of water quality interventions on diarrhea. We find plausible theories grounded in the available evidence for both believing that water quality interventions reduce diarrhea and for the more pessimistic conclusion that these interventions do not have an effect.

Based on the ambiguity of the results, and the fact that it’s plausible the interventions on their own do not have an effect, we currently do not consider water quality interventions to be evidence-backed and do not include them among our priority programs. This doesn’t mean we’re confident that the interventions don’t work, but it does mean that we’re not confident that they do.

Note that Dr. Alix Zwane, formerly the Executive Director of Evidence Action (which runs one of our top charities, the Deworm the World Initiative, in addition to a charity which conducts water quality programs, Dispensers for Safe Water) and now CEO of the Global Innovation Fund, disagrees with our conclusion and has a more positive view of the benefits of water quality intervention. Her comments on our report, along with comments by Thomas Clasen, one of the authors of the Cochrane report, can be found here.

Comparison to other programs we recommend

While the evidence for water quality interventions is mixed, it is stronger than for most other interventions that we’ve come across. Depending on what you believe about the weaknesses of the evidence base, it’s arguably as good as the evidence for deworming, which two of our top charities focus on. Deworming also has a complicated evidence base which we have written about extensively, most recently here.

Our initial estimates, however, found that water quality interventions wouldn’t be as cost-effective as deworming programs, and so recently we have not prioritized further research into water quality charities.

That said, because this evidence base is stronger than most, it’s possible that a strong water quality charity could qualify as one of our standout charities, even though we would expect to still recommend our current top charities more highly. A strong water quality charity could even one day become one of our top charity recommendations, especially if our current top charities run out of room for more funding.

Comments

  • Philip Dahlin on May 4, 2016 at 10:26 am said:

    Considering the weak evidence with water-alone based interventions, have there been any randomized-controlled studies of unified WASH (water, sanitation and hygiene) programs? I know people usually avoid undertaking multiple intervention studies due to the intricacies of trying to measure which of the interventions was most effective, but because of the problems mentioned in the blog post, it seems to me that a full WASH approach is needed to achieve significant impacts. Any studies to point to? Any undergoing that you know of?

  • Sean on May 4, 2016 at 8:34 pm said:

    Hi Philip – our understanding is that there’s an ongoing study, WASH Benefits (http://www.washbenefits.net/objectives.html), that’s measuring the effects of a combined water, sanitation and handwashing program. We’re not sure what its current status is.

  • Ian Turner on May 9, 2016 at 9:02 am said:

    Can you clarify if “water quality interventions” includes wells or pumps, or are you thinking just of water treatment?

  • Sean on May 9, 2016 at 5:47 pm said:

    Hi Ian – thanks for asking! This write-up just refers to water treatment programs that aim to remove or disable the pathogens found in water (and not to wells, pumps, or other programs that increase access to water).

  • Hauke Hillebrandt on May 26, 2016 at 4:47 pm said:

    I believe there might be an statistical error here, in that you have a significant effect (unblinded studies) and a nonsignificant effect (blinded studies), but your interpretation above suggests that this difference itself is significant, even though that would require a statistical test (looking at the interaction). Especially because the p-value is somewhat lowish for the one-tailed test, the differences in significance might not be significant (but they might be!). (This is a common statistical error in the academic literature- see http://www.nature.com/neuro/journal/v14/n9/full/nn.2886.html). If the difference is not significant, the interpretation might be quite different, as one couldn’t argue that the unblinded trials are systematically confounded. This might justify looking at the overall meta analysis by Cochrane, especially given that the unblinded studies were downgraded and there might not be any more blinded RCTs due to ethical issues in the near future (http://ehp.niehs.nih.gov/wp-content/uploads/advpub/2015/12/ehp.1510532.acco.pdf).

  • Paul Garner on May 27, 2016 at 9:47 am said:

    Givewell. You are basing your judgements on a Cochrane review published in 2006. This has been substantially revised in 2015. We suggest you revisit your appraisal and draw on this review.
    http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD004794.pub3/abstract

    Best wishes
    Paul

  • Luke on May 29, 2016 at 5:41 pm said:

    Hi Sean
    Thanks for the very interesting article. As you (and Philip) suggest, multiple intervention may be more effective, although it may be difficult to assess the benefits of individual elements of a broader program. It reminds me that the more we focus on the ‘micro’, the more we lose sight of the bigger picture with regard to health, or anything else for that matter. At the risk of being howled down by the statisticians, may I suggest that trying to measure the ‘success’ of water quality interventions may be a waste of time. The real measures of success in a country with poor water quality or a poor measure of any other ‘standard of living’ measure is an increase in education and empowerment of people within that society. If we can support the move towards those, then water quality and other problems will be solved by the people themselves.

  • Sean on May 31, 2016 at 1:17 pm said:

    Hi Hauke – thanks for the comment! I agree that we sometimes sound like we’re treating the difference between the blinded and unblinded results as though it’s statistically significant, and we should avoid that, since we haven’t tested it. But, whether or not the results are significantly different isn’t crucial to our bottom line, since our point is that we should potentially just ignore the unblinded studies completely.

    We’re planning on having someone take a closer look at the data at some point to determine whether the results are significantly different, and to verify that our bottom line on the intervention should remain the same.

  • Sean on June 1, 2016 at 2:04 pm said:

    Hi Paul, thank you for sharing this. We hadn’t seen this update, so thank you for bringing it to our attention.

    I looked at the update relatively briefly, and my guess is that it’s unlikely to change our views because the blinded studies under consideration remain the same as the ones that we considered for our full report (except Rodrigo 2011, but we would exclude it from that report because we focus there on studies in developing countries).

    We are planning to revisit our assessment of this intervention in the future, and when we do, we’ll take a closer look at this new Cochrane review.

  • Jacob on June 7, 2016 at 1:31 pm said:

    It would be interesting to compare of water quality interventions to, say, oral rehydration therapy education programs when it comes to reducing the death + disability burden of diarrheal diseases. I notice ORT is on your priority programs list — any plans for an investigation?

  • Sean on June 8, 2016 at 3:37 pm said:

    Hi Jacob – we don’t have any plans for such an investigation at the moment. One major question would be whether oral rehydration programs have significant room for more funding – whether there are places where funding is the bottleneck to ORT being available when necessary.

  • User on June 9, 2016 at 12:45 pm said:

    I accept that GW’s evidence-based approach to
    its choice of humanitarian programs to support
    eliminates water quality (health-related) programs, at present.

    I plead that GW devote serious attention to
    the crucial matter of water SUPPLY
    (especially residential/community supply programs;
    not necessarily to exclusion of supply programs that
    benefit small agricultural/business uses).

    It should be easy to document quality-of-life differences
    between having water and not having it
    (or between having one gallon and having five gallons).

    [A GW staffer posted this comment on behalf of a user who was unable to post it themselves due to technical issues.]

  • Sean on June 9, 2016 at 12:46 pm said:

    User – Thanks for the thoughts. We haven’t prioritized research on water supply interventions because our impression is that there isn’t clear evidence supporting their health impacts, and charities we’ve talked to haven’t been able to show us strong monitoring to demonstrate that their programs have increased access to water.

Comments are closed.