Carrying out a highly rigorous study is difficult, expensive, and can take a long time. It’s very rare to see a charity that has carried out a highly rigorous study of its own work – one that can isolate the impact of its program from all the other factors affecting its clients. As such, we generally recommended charities running programs that have independently been rigorously studied, and this approach leads to major challenges in generalizing.
- Against Malaria Foundation distributes bednets, which have been found to save lives and reduce malaria in multiple high-quality studies – but all of these studies were carried out over 10 years ago, when insecticide resistance was less of a concern and general health was likely worse (leading to possibly elevated impacts for nets), and the programs studied were different in some important ways from a typical modern net distribution.
- Schistosomiasis Control Initiative and Deworm the World Initiative carry out deworming, a program whose case for effectiveness rests on two studies: one on a hookworm eradication campaign in the American South in the early 20th century, and another from a randomized controlled trial in Kenya during El Nino (in which El Nino led to greatly elevated levels of worm infection, and thus likely greatly elevated impact of treatment).
- Previously, our case for cash transfers rested on a variety of studies looking at programs that were structured differently – in important ways – from GiveDirectly’s program. In particular, they often involved conditional cash transfers (transfers given in exchange for e.g. sending children to school and engaging in other donor-desired behavior) and generally involved smaller amounts given over longer periods over time.
This year, for the first time, we’re aware of a charity that can point to a highly rigorous and recent study of its own work. The charity is GiveDirectly and the study is Haushofer and Shapiro 2013, a randomized controlled trial (with a pre-analysis plan to boot) reporting:
- Increased consumption across a variety of categories, including food, education, business expenses and durable assets, but not alcohol and tobacco.
- Improved standing on a food security index (which was one of the topmost outcomes of interest according to the pre-analysis plan) and an index of psychological well-being.
- An (unelaborated) claim in a recently updated report on the study that spending on iron roofs produced a return on investment of 23%.
There are, as with any study, major limitations.
- The majority of recipients received smaller transfers than GiveDirectly’s standard program provides (though some received roughly the same size transfer).
- Much of the data is based on self-reports (though this is also true of other studies, such as those supporting deworming, that look at income and consumption).
- The study only covers the area GiveDirectly was working in at the time; as GiveDirectly expands, it will encounter substantially different environments.
- The study is relatively short-term, and we hope there will be follow-ups (we know there are some plans on this front).
The Economist has also discussed this study.
Overall, this study represents a substantial update on GiveDirectly, and puts it in something of a unique position relative to other charities we’re aware of.
It seems to us that much of our audience overestimates the evidence base for deworming and underestimates the evidence base for cash transfers. In our view, GiveDirectly has – all things considered – the most strongly evidence-backed intervention of our current three top charities (though we’d still say the case for bednets is stronger).