When we started researching water quality interventions, we originally guessed that there was clear evidence that the programs were effective at improving people’s health based on our general intuitions about the programs and our initial read of the relevant Cochrane review’s bottom line. Once we dug into the details of the research, though, we realized that the evidence regarding the program is mixed — a major reason that we don’t currently recommend charities that work on improving water quality.

This post describes that research, and is drawn from our full report on water quality interventions, which provides more detail and citations. The goal of this blog post is to summarize that report in a more accessible form.

The key points of this post:

• We formed an initial impression that water quality interventions (e.g., chlorine tablets or dispensers; water filters; other types of treatment that clean drinking water in the developing world) improve people’s health, based on our understanding of the dangers of contaminated water. However, digging into the evidence regarding clean water led us to conclude that the evidence base is mixed.
• We’re not confident that water quality interventions are not effective; rather, we believe that it’s unclear whether or not they’re effective, and so we don’t include water quality among our priority programs.

The intuitive case for water quality charities

Intuitively, providing a method for cleaning water seems like it should be an effective way to improve health. There is little debate that contaminated water causes diarrhea or that the methods used to treat water reduce contamination, and so it’s reasonable to expect that interventions that reduce water contamination ought to reduce diarrhea. And because diarrhea is deadly – the Global Burden of Disease estimated that in 2010, diarrheal disease caused about 10% of child deaths in developing countries – these interventions could be potentially life-saving.

Developing world randomized controlled trials

A number of independent studies have been done on water quality programs. The results of those studies look very promising: They largely find that providing clean water reduces diarrhea rates.

Specifically, 20 randomized controlled trials conducted in developing countries with around 30,000 participants found that interventions to improve water quality were effective at preventing diarrhea, with an average 35% reduction in the odds of getting diarrhea, according to a meta-analysis (study of studies) conducted by the Cochrane Collaboration. Relative to most interventions we’ve considered, twenty randomized controlled trials is an unusually high number of studies.

We typically consider randomized controlled trials to be a particularly strong method of studying global health outcomes. These studies compare a group that receives an intervention – in this case, some method for cleaning water, like chlorine tablets – with a control group that doesn’t. The intervention and control groups are randomly chosen to try to ensure that two groups are similar in every way other than whether or not they receive the intervention. This way, any differences between the groups ought to be due to the intervention.

The Cochrane Collaboration’s findings, then, strongly suggest that the cleaner water is responsible for the fall in diarrhea rates.

Possible weakness of the randomized controlled trials

When we examined these studies more closely, we noticed a potential weakness. As the Cochrane review states: “Only four of the 22 randomised controlled studies, however, were properly blinded”. (22 because the Cochrane review also considers 2 studies conducted in the United States, which we excluded from our analysis.) In other words, the researchers conducting these studies measured diarrhea rates by asking participants to report them, and the study participants knew whether or not they had received the intervention that was being tested.

It’s possible that this knowledge influenced how participants reported diarrhea cases. For example, participants who knew that their water was being treated may have been less likely to report cases of diarrhea, perhaps because they recalled using the treatment and misremembered how many cases they had, or because they anticipated what the surveyor wanted to hear. If that were the case, the study would find an inaccurately low number of diarrhea cases among people that received the clean water intervention, which would result in the study finding an inflated effect of the intervention.

To deal with the issue of participants knowing whether or not they received the intervention (and the possibility that this influences reporting), studies can use “blinding” to disguise who has received the intervention by distributing a placebo (for example, a fake, non-functional water filter) to the control group. When this method is used, participants don’t know whether they have the fake filter or the real one, and so any bias in how they report their cases of diarrhea ought to affect both groups equally, so we can still measure the difference between the two groups without this source of bias affecting the outcome.

Blinded randomized controlled trials should let us know whether the tested intervention is actually improving people’s health, or whether other sources of harmful pathogens are negating any positive effect of the clean water programs. If the latter were true, we may not expect supporting the intervention to have a beneficial impact on the population.

Blinded studies

We identified five blinded studies of water quality interventions that have been conducted in developing countries (which includes some published after the Cochrane review). These studies measured household interventions to clean water; four studied chlorination and one examined the impact of water filters. They did not find a statistically significant effect of the interventions on diarrhea. (Pooling the effects of the studies, we calculated a 5% reduction in the percent of days with diarrhea for children under 5, with a 95% confidence interval ranging from a 6% increase to a 14% decrease, and no reduction for all ages with a 95% confidence interval ranging from a 12% increase to an 11% decrease.)

This suggests that biased reporting from study participants who knew whether or not they were receiving an intervention may explain why the non-blinded studies in the Cochrane review found a statistically significant effect – that is, it suggests that people in the non-blinded studies may have underreported diarrhea rates when they knew that they were receiving a treatment, but when people were unsure whether or not they were receiving a treatment, there was no reporting bias and so no statistically significant effect was found.

Possible weaknesses of the blinded studies

However, these blinded studies have their own weaknesses. In one, the fake filter used for the control group was actually somewhat effective at cleaning the water, meaning that the control group (which wasn’t supposed to be receiving any clean-water benefits) received water that was at least partially cleaned. It’s not surprising, then, that the study didn’t find a noticeable difference between the diarrhea rates of the two groups.

Two of the other blinded studies were conducted in groups that already had low levels of diarrheal disease, which may have made it harder to detect a difference between the control and intervention groups. The other two had small sample sizes, of 112 (the smallest included in the Cochrane review) and 287 people. (Additional problems with the blinded studies are discussed here.)

These problems suggest that the blinded studies are not conclusive in determining that there’s no statistically significant effect of providing chlorine and water filters to reduce diarrhea rates – each of those problems might explain why the blinded studies found no statistically significant effect, even if the intervention itself is actually beneficial.

(There are a handful of other, different studies that we do not find compelling – you can read more about those here.)

How could these interventions fail to reduce diarrhea rates?

Considering the scientific consensus that contaminated water can cause diarrhea, one could reasonably ask whether it’s even plausible that water quality interventions have small or negligible effects (as opposed to large effects). Is it? There is a potential explanation: Other pathways for pathogens, such as from fecal matter on unwashed hands or from food via flies or other people’s hands, especially in an environment with poor sanitation, may cause a significant number of diarrhea cases, so it’s possible that the gains from simply providing clean water could be small if people are regularly infected via other sources. In fact, it’s possible that water quality interventions might have little effect on diarrhea unless all sources of pathogens are eliminated. The possibility that providing clean water will have little effect in an environment with other sources of pathogens is especially concerning because the blinded studies found no statistically significant effect of the intervention.

Bottom line

We concluded in our intervention report (emphasis included for this post),

The strength of the evidence relies on how one chooses to weigh a few, blinded studies with some methodological weaknesses that report no effect against a large number of unblinded studies, some of which may also have some methodological weaknesses, that, on average, report large effects… Overall, we are ambivalent about the effect of water quality interventions on diarrhea. We find plausible theories grounded in the available evidence for both believing that water quality interventions reduce diarrhea and for the more pessimistic conclusion that these interventions do not have an effect.

Based on the ambiguity of the results, and the fact that it’s plausible the interventions on their own do not have an effect, we currently do not consider water quality interventions to be evidence-backed and do not include them among our priority programs. This doesn’t mean we’re confident that the interventions don’t work, but it does mean that we’re not confident that they do.

Note that Dr. Alix Zwane, formerly the Executive Director of Evidence Action (which runs one of our top charities, the Deworm the World Initiative, in addition to a charity which conducts water quality programs, Dispensers for Safe Water) and now CEO of the Global Innovation Fund, disagrees with our conclusion and has a more positive view of the benefits of water quality intervention. Her comments on our report, along with comments by Thomas Clasen, one of the authors of the Cochrane report, can be found here.

Comparison to other programs we recommend

While the evidence for water quality interventions is mixed, it is stronger than for most other interventions that we’ve come across. Depending on what you believe about the weaknesses of the evidence base, it’s arguably as good as the evidence for deworming, which two of our top charities focus on. Deworming also has a complicated evidence base which we have written about extensively, most recently here.

Our initial estimates, however, found that water quality interventions wouldn’t be as cost-effective as deworming programs, and so recently we have not prioritized further research into water quality charities.

That said, because this evidence base is stronger than most, it’s possible that a strong water quality charity could qualify as one of our standout charities, even though we would expect to still recommend our current top charities more highly. A strong water quality charity could even one day become one of our top charity recommendations, especially if our current top charities run out of room for more funding.