# The GiveWell Blog

On June 22, we held a virtual event on research into water quality interventions, featuring presentations from Michael Kremer of the University of Chicago’s Development Innovation Lab; Brett Sedgewick from Evidence Action, the parent organization of Dispensers for Safe Water; and Stephan Guyenet, Elie Hassenfeld, and Catherine Hollander of GiveWell. (If you weren’t able to attend, we’ve published a video recording, audio recording, and transcript here.)

## On our meta-analysis of water treatment’s effects

Q: Michael Kremer and his team found a very large effect of water treatments like chlorination on child mortality, and GiveWell found a smaller but still significant mortality effect. What about effects beyond mortality? For instance, I’m curious about the possibility of increasing the child’s adult income because of improved overall health, or the impact of reducing the family’s expenditure on medical treatment, etc. How confident can we be in these results?

A: The single largest benefit we model from chlorination interventions like Dispensers for Safe Water is a reduction in mortality among children under five, but we do incorporate other benefits as well:

• Development effects. We believe reducing illness in children probably improves their development, leading to slightly higher income in adulthood. We refer to this as “development effects” and include it in our model. We don’t have direct evidence that water treatment causes this, but we do have direct evidence that malaria prevention and deworming cause it, and we think it probably applies here as well.
• Medical costs averted. If a child becomes ill less often because their water is chlorinated, we do expect the family will save money on medical treatment, and we factor in this savings. The estimate we arrived at in our cost-effectiveness analysis for water treatment was higher than we’d initially expected. We’re working on refining this and investigating how generalizable it might be to other types of programs.
• Over-five mortality. We estimate that water treatment reduces mortality in people older than five by 1 to 4%. That’s less than the 6 to 11% reduction in all-cause mortality we estimate in children under five, but not negligible, given that there are more people over five than under five. Much of this benefit is expected to occur in children just over five.[4]

Q: In describing the meta-analysis on which this grant recommendation was based, you mentioned that you took steps to limit publication bias, and Michael Kremer also mentioned doing checks for publication bias in his own analysis. Can you explain what that is and why it mattered to this investigation?

A: “Publication bias” is when a body of scientific literature is biased because certain types of results are more likely to be published than others. There are different types of publication bias, but a common example is that studies that produce statistically significant results are more likely to be published than studies that do not produce such results. This biases the available literature on the subject in favor of research that finds an effect, which causes the literature as a whole to exaggerate effect sizes. The Wikipedia page on publication bias has some good further explanation and examples.

When reviewing the evidence behind an intervention, we consider whether publication bias is a factor and make adjustments in our cost-effectiveness models accordingly. In this case, we decided to do our own meta-analysis to arrive at our own pooled estimate of water chlorination’s impact on deaths. We think our pooled estimate, which guided our decision to recommend the grant to Dispensers for Safe Water, is resistant to publication bias because the vast majority of the weight comes from three large, recent water quality trials. We think this because all of the recent large trials of water quality have reported mortality outcomes, and results from large trials usually don’t go unpublished, regardless of outcome (so it is unlikely that only statistically significant results showing a benefit would be chosen for publication). We also excluded smaller trials with shorter follow-up periods (that is, the period of time in which study participants are observed after the treatment), which tend to be more susceptible to publication bias.[5]

We ultimately arrived at an estimated 14% reduction in all-cause mortality in children under five for chlorination interventions generally, with lower estimates for the effect of Dispensers for Safe Water specifically. This led us to update our cost-effectiveness estimate for Dispensers for Safe Water to between four and nine times as cost-effective as unconditional cash transfers (depending on location). You can read more about Michael Kremer et al.’s meta-analysis and our own meta-analysis here, and more about how we arrived at our cost-effectiveness estimates here.

Q: Michael Kremer mentioned that his team’s analysis included Bayesian estimates. I know just a little about Bayesian analysis in RCTs, but I recall the idea of enthusiastic and skeptical priors. Were those used in this Bayesian analysis? And, are there “enthusiasts” and “skeptics” on this issue?

Kremer et al. used a prior centered around zero effect, with a wide standard deviation. As they explain in the supplementary materials for their paper: “For τ, we set a normal distribution with mean 0 and standard deviation of 10. This prior encodes the belief that causal effects should not be thought of as large unless data contains evidence to the contrary” (p. 3). Kremer et al. conducted a number of sensitivity analyses, but we are not aware that they tried pessimistic and optimistic priors in their Bayesian meta-analysis.[6]

More broadly, GiveWell did not consider optimistic and pessimistic scenarios. We sometimes consider these for decision-making purposes if a program is below our funding bar and we want to see how plausible it is that further work would raise it above the bar. However, if a program is over our funding bar, we typically make funding decisions based on our best guess, assuming that the best guess of cost-effectiveness represents the expected value of the entire probability distribution of possible outcomes. In other words, we assume that the probability that the program is better than we think balances out the probability that it is worse than we think. We are aware that this assumption may not always be satisfied, and we may do further work on this issue.

Michael Kremer’s meta-analysis is recent and not yet published in a peer-reviewed journal, and we aren’t aware that people have publicly aligned themselves as advocates or skeptics yet. However, the external expert reviewers who we contracted to review an earlier draft of the Kremer et al. meta-analysis were fairly skeptical of the findings.[7] Our analysis of the data reflects the reviewers’ skepticism, and our estimated effect size is quite a bit smaller than the one in the Kremer et al. meta-analysis (although confidence intervals overlap),[8] so it would be fair to say we are also on the more skeptical side.

## On unintended consequences of charity-run programs

Q: What do you make of the criticism that charities reduce the pressure put on local governments to deliver better services?

A: This is a reasonable critique. We believe that there could be knock-on effects from funding charities to implement water chlorination (or other) programs, such as making governments less likely to provide these services themselves, and right now we are not very confident in our ability to measure the likelihood of such effects. However, we believe there is a role for private funding to play in getting vital services to people who need them. Water treatment is a service that is clearly needed and not currently provided by the governments where Dispensers for Safe Water operates.[9] Based on our conversations with Evidence Action and others, it also appears to be highly neglected by private funders, and therefore well suited for impact-motivated philanthropy to step in.

Additionally, many programs are set up to be partnerships between government and non-governmental organizations, in which the NGO provides training, monitoring and evaluation activities, or other forms of “technical assistance” to government staff, who actually deliver the services. We’ve found that these partnerships can result in more impact than NGOs acting alone.

For example, Deworm the World Initiative supports the governments of India, Kenya, Nigeria, and Pakistan in their implementation of deworming campaigns (more here). Against Malaria Foundation works with countries’ national malaria programs to determine funding needs and allocation decisions for insecticide treated nets campaigns (more here).

Beyond our top charities, we’ve recommended grants to organizations that provide technical assistance to governments to make their programs more effective. For example, we recommended a grant to another program from Evidence Action that is working to support the Liberian government in switching from HIV rapid tests during routine antenatal care to more effective dual HIV/syphilis rapid tests and syphilis treatment (more here).

## On GiveWell’s research process and sources of potential new funding opportunities

Q: Could you please share the process GiveWell uses to make funding decisions—how you narrow down potential projects for investigation and then grantmaking?

A: Our process for investigating a grant consists of three stages, though not all of them apply to all grants:

• Research review. We look at the evidence behind an intervention, talk to experts and possible implementing organizations, and build a rough cost-effectiveness model. This stage applies more to new interventions than top charities, since we already are very familiar with top-charity programs and have directed funds to them before.
• Strong interest. We’ve determined that the program likely is cost-effective enough for us to consider funding, and we want to move on to a deeper investigation of a specific grant opportunity.
• Grant recommendation (called “conditional approval” internally, as the grant approval is conditional on the funder’s confirmation). We’ve done a thorough analysis of cost-effectiveness and spent a lot of time talking to the potential grantee about the funding opportunity. We’ve decided it meets our criteria for funding, and we want to recommend the grant (or make it ourselves).

When evaluating a program for funding, we look for evidence of its effectiveness, its impact per dollar spent, room for more funding (i.e., how much money it can productively use), and whether the organization seems like a strong partner that will share transparently with us and allow us to share our views about it publicly. We also consider what effect our funding in this situation will have on other funders’ decisions—i.e., whether it will cause them to allocate more or less money to this intervention (more here).

Right now we have 350 programs in our pipeline that we’d potentially like to investigate, and we’re actively investigating about 60 of them. But we don’t expect to make grants to all of them—we think probably about 15 of these active investigations will result in grants.

You can read more about our research process and see a partial list of prioritized programs here.