Carrying out a highly rigorous study is difficult, expensive, and can take a long time. It’s very rare to see a charity that has carried out a highly rigorous study of its own work – one that can isolate the impact of its program from all the other factors affecting its clients. As such, we generally recommended charities running programs that have independently been rigorously studied, and this approach leads to major challenges in generalizing.
- Against Malaria Foundation distributes bednets, which have been found to save lives and reduce malaria in multiple high-quality studies – but all of these studies were carried out over 10 years ago, when insecticide resistance was less of a concern and general health was likely worse (leading to possibly elevated impacts for nets), and the programs studied were different in some important ways from a typical modern net distribution.
- Schistosomiasis Control Initiative and Deworm the World Initiative carry out deworming, a program whose case for effectiveness rests on two studies: one on a hookworm eradication campaign in the American South in the early 20th century, and another from a randomized controlled trial in Kenya during El Nino (in which El Nino led to greatly elevated levels of worm infection, and thus likely greatly elevated impact of treatment).
- Previously, our case for cash transfers rested on a variety of studies looking at programs that were structured differently – in important ways – from GiveDirectly’s program. In particular, they often involved conditional cash transfers (transfers given in exchange for e.g. sending children to school and engaging in other donor-desired behavior) and generally involved smaller amounts given over longer periods over time.
This year, for the first time, we’re aware of a charity that can point to a highly rigorous and recent study of its own work. The charity is GiveDirectly and the study is Haushofer and Shapiro 2013, a randomized controlled trial (with a pre-analysis plan to boot) reporting:
- Increased consumption across a variety of categories, including food, education, business expenses and durable assets, but not alcohol and tobacco.
- Improved standing on a food security index (which was one of the topmost outcomes of interest according to the pre-analysis plan) and an index of psychological well-being.
- An (unelaborated) claim in a recently updated report on the study that spending on iron roofs produced a return on investment of 23%.
There are, as with any study, major limitations.
- The majority of recipients received smaller transfers than GiveDirectly’s standard program provides (though some received roughly the same size transfer).
- Much of the data is based on self-reports (though this is also true of other studies, such as those supporting deworming, that look at income and consumption).
- The study only covers the area GiveDirectly was working in at the time; as GiveDirectly expands, it will encounter substantially different environments.
- The study is relatively short-term, and we hope there will be follow-ups (we know there are some plans on this front).
The Economist has also discussed this study.
Overall, this study represents a substantial update on GiveDirectly, and puts it in something of a unique position relative to other charities we’re aware of.
It seems to us that much of our audience overestimates the evidence base for deworming and underestimates the evidence base for cash transfers. In our view, GiveDirectly has – all things considered – the most strongly evidence-backed intervention of our current three top charities (though we’d still say the case for bednets is stronger).
Comments
Interesting. Thank you for sharing this level of detail.
I see you still consider evidence for bednets being the strongest, yet, following the issues with AMF, you are not recommending any charity in this area.
Can you clarify if this is because you are confident that no other bednet charity fulfills your other requirements or do you think there are other promising charities in this area, but you just didn’t have the time to fully evaluate them yet (given the recency of AMF’s problems).
If it’s the latter, do you expect to recommend a bednet distributing charity soon and would it even make sense to wait for this before giving to one of your current top charities?
The Economist article cited, while applauding unconditional cash transfers (UCTs) such as GiveDirectly, had a 2nd main point: that conditional transfers (CCTs) work even better in many ways. Eg: They seem to help more people per $. There is more evidence supporting them. It points to a study by Sarah Baird that studied 35 UCT and CCT programs and showed some better outcomes for CCTs. The Economist concludes that while UCTs are great, CCTs are even better.
Thoughts? Has GiveWell considered any CCT-based charities alongside GiveDirectly? Or are most CCTs government-based and thus there aren’t easy charities to contribute to as US donors?
Karl, we agree with the conclusion that conditional cash transfers are likely to be more effective in terms of the specific aspect that they choose to condition on, but we’re less confident that those conditioned outcomes are leading to long term reductions in poverty than the Economist article is. For instance, see this CGD blog post: http://www.cgdev.org/blog/cash-transfers-and-deeper-causes-poverty
That said, we are interested in finding organizations that focus on CCTs. One that we’re watching is New Incentives. (Conversation notes here: https://www.givewell.org/files/conversations/New%20Incentives%2012-13-12%20(public).pdf)
I don’t know of any (other) sizable organizations that focus primarily on CCTs, and New Incentives itself is quite small.
If the case for bednets is stronger, is there room for another charity doing bednets to come forward? Is the problem that other non-AMF bednet charities that they’re not set up to provide the kind of data that indicates they’re performing in the same neighborhood of effectiveness as AMF?
(I feel like this may have been addressed somewhere, but if so I missed it.)
Julia- I addressed this to some extent in my answer to Eric here.
Cash transfers sound like an intervention where the results and analysis of a trial could miss some effects observed in a full roll out. This is of course a common criticism of trials. Does scale up create any administrative burden for local governments that would compromise provision of other services?
Chris – I think the concern that trials “could miss some effects observed in a full roll out” is a reasonable one. We discussed some aspects of it with Paul Niehaus in this conversation. I don’t think it’s likely that GiveDirectly is creating a burden on local governments that reduces their provision of other services, because local governments don’t play a huge role in their process.
Comments are closed.