GiveWell’s 3 full-time staff (myself, Elie Hassenfeld and Natalie Stone) will be living and working in Mumbai from mid-August through late November.
Developing-world aid has become a major focus for us, and we hope to have more opportunities to see aid work up close (along the lines of my trip to Africa earlier this year).
Please let us know if you have suggestions for charities we should visit, contacts at such charities, or any other advice/suggestions/contacts.
USAID’s most recent report on microfinance and microenterprise development tells an interesting story and, in my view, shows just how widely microfinance has been (and continues to be) misunderstood. While many advocate that microfinance institutions focus on people under the global “extreme poverty line”, USAID’s report implies that actually doing so is rare and even unrealistic.
Background: the myth of targeting the poorest
The international “extreme poverty line” is around the equivalent of US$1.25 per day, and around 1.4 billion people worldwide (and over half of those in sub-Saharan Africa) are estimated to live below this line (see the discussion in our international aid report).
Many seem to believe that people in this category are appropriate – even ideal – as clients. For example, see Opportunity International and Grameen Foundation stressing the need to reach the “poorest” and “most vulnerable.” Both Accion and CGAP cite the entire 3-billion-strong set of people under the US$2/day line as potential microfinance clients (upwards of 50% of this set falls below the “extreme poverty line”).
U.S. official aid seems to have taken this idea particularly far. For the past several years, USAID has been required by law to target the “very poor,” defined partly with reference to this “extreme poverty line”:
Both the Microenterprise for Self Reliance Act of 2000 (henceforth, the 2000 Act) and the MRAA mandate that at least half of all USAID funding for microenterprise development directly benefit the very poor. The 2000 Act initially defined the “very poor” as the bottom [poorest] half of those living below each country’s national poverty line … Subsequent amendments to the 2000 Act mandated a second, much more ambitious approach … the amended law created a second definition of the “very poor” — those living on less than the equivalent of $1 per day, calculated using purchasing power parity (PPP) exchange rates. The law made clear that, for any given country, the applicable definition of the very poor would be the more inclusive one.
(Note that “$1/day” may be a reference to the $1.25/day “extreme poverty line” discussed here – see note 6.)
Investigating actual poverty levels of clients
To its credit, USAID put significant effort into tracking whether it was actually meeting this goal, developing poverty assessment tools for assessing clients’ poverty levels and requiring certain grantees to use them. The results:
Among the eight microfinance institutions that applied and reported on the Poverty Assessment Tools, the average share of Funds Benefiting the Very Poor (FVP) is estimated at 28.5 percent, up from 16.3 percent in FY 2007. … For the 14 enterprise development programs that applied and reported on the Poverty Assessment Tools, average FVP is estimated at 26.0 percent, up from 20.5 percent in FY 2007 …
USAID did not come close to its target of 50% “extremely poor” clients. Furthermore, it concluded that continuing to push for this target would be unwise:
As matters stand, USAID sees no promising options for meeting the FVP target. It cannot do so by reallocating funds among its existing partners, because with the exception of one small program, none had more than 50 percent “very poor” clients. It cannot do so by shifting funds to established microenterprise organizations that are not already receiving USAID funding, because few if any such organizations are voluntarily applying the USAID-certified poverty assessment tools, and no such organization has offered solid evidence that it has more than 50 percent “very poor” clients …
Misguided target?
USAID does not conclude that microfinance/microenterprise projects should be de-emphasized (it observes that “the great majority of clients … are very poor, at least in commonly used terms”). Instead, it concludes that the idea of serving the poorest was unrealistic/inappropriate in the first place.
the overall pattern of results lend further weight to the point that USAID raised in last year’s Annual Report – that current law imposes too low a threshold for being “very poor.” This very narrow definition makes it impossible for USAID to allocate its microfinance and microenterprise funding so as to reach the legislative target of directing 50 percent of the benefits of microenterprise funding to the “very poor,” without undermining other goals emphasized in the same legislation, such as sustainability and support for broad-based economic growth.
Unfortunately, this definition of being “very poor” was adopted without any evidence that a 50 percent FVP target based on this definition could be reached. Two years of results using the poverty assessment tools strongly suggest that the target cannot be reached without inflicting undesirable side effects on sustainability and economic development. In short, USAID sees no realistic prospect of reaching the target contained in the law, and urges prompt and serious consideration of changes in the law. (Bold mine; italics in original)
I’m inclined to agree with USAID’s conclusion. I agree that people with incomes well above the “extreme poverty line” can still be very poor, certainly poor enough that I’m interested in donating to help them. So my point is not that microfinance is being carried out inappropriately, or is failing to reach the very poor.
Rather, I’m noting yet another way in which microfinance seems to have been badly misunderstood by its biggest funders and proponents. USAID, and by implication its grantees, seem to have thought that they were serving the world’s poorest – to the point of legislating it – without any data, and wrongly. It’s another debunked myth, and another sign that the funding and the stories have gotten ahead of the facts.
Providing clean water to people living in developing countries is a cause that many donors are interested in. Among other hardships, unclean water can lead to diseases such as diarrhea, which is responsible for millions of child deaths annually. Unfortunately, we have found little evidence that charities’ efforts to improve water infrastructure in the developing world have resulted in decreased incidence of diarrhea and we have not identified a water charity we can confidently recommend to donors.
A recent randomized controlled trial (RCT) Spring cleaning: A Randomized Evaluation of Source Water Quality Improvement, by J-PAL researchers, however, provides the first strong evidence we’ve seen for the impact of a water infrastructure project. Unprotected springs in western Kenya were randomly assigned to receive “protection” at the beginning or the end of the study period. “Protection” involved encasing the spring in concrete and directing water through a pipe so that water did not come into contact with contaminants on the ground. The researchers found that children under the age of 3 living in households that used protected springs were 25% less likely to have had an episode of diarrhea in the last week than children in households that used unprotected springs.
Note that an earlier draft of this study found no statistically significant impact on diarrhea incidence, so the update is a new and somewhat surprising piece of evidence.
So the project appears to have worked; what about the cost-effectiveness? This is tricky to assess:
They used a different approach than ours, and having considered it, we defer to their estimate as likely more accurate.
We believe the source of the difference in our cost-effectiveness estimates is due to a difference in the conversion calculation used to translate between cases, DALYs, and deaths averted. The study’s authors used a conversion factor of 864 cases averted = 32 DALYs averted = 1 death averted, based on data on child deaths from diarrhea in sub-Saharan Africa from from the Global Burden of Disease report (2006, data from 2001) and data on cases of diarrhea per child in sub-Saharan Africa from Kirkwood (1991) (Pg 21). We used a conversion factor of 2136 cases averted = 67 DALYs averted = 1 death averted, based on data on total deaths, DALYs, and cases lost due to diarrhea from the World Health Organization for 2004. As we now believe that the study’s authors made more reasonable assumptions than we did, we defer to their estimate and conclude that, under conditions similar to the ones in the study, spring protection is may be a highly cost-effective way to save lives in sub-Saharan Africa.
Regardless, this study is good news for donors interested in improving health through water infrastructure. Independent evidence of effectiveness lowers the burden of proof somewhat for a charity conducting spring protection. Still, we feel the burden of proof for such a charity remains higher than for one delivering simple, proven medical interventions, such as vaccines or treatments for tuberculosis, malaria, or intestinal worms. Here’s why:
Four out of the eleven charities listed at our overview of water programs appear to do at least some spring protection, though none appear to focus on spring protection.* In investigating a charity working on spring protection, we would want to see evidence that the springs are monitored over time to determine whether springs remain in use and in good condition. Ideally, we would also like to see that disease rates have declined among users of protected springs in relation to a suitable comparison group. From what we’ve seen to date, this is a bar that water charities have yet to reach.
*Specifically:
There are a lot of reasons to be interested in charities focused on neglected tropical diseases (NTDs), and particularly on deworming schoolchildren.
We are positive on deworming as an intervention. But we feel that for individual donors, it is necessary to form confidence in charities, not just interventions. This is where our interest in NTD control hits major obstacles. This post goes through our questions and the difficulty we’ve had answering them, and focuses on charities focusing partly or fully on deworming.
Key questions
As far as we can tell (details below), the major organizations focused on deworming work heavily with developing-world governments. They spend money on both advocacy and subsidies for government control programs. We therefore have the following questions:
Schistosomiasis Control Initiative (SCI)
A key roadblock to evaluating SCI is that it does not share any information about its budget by program (past or planned). Last year, after failing to find this information on its website, we contacted Prof. Alan Fenwick, and after significant back-and-forth we were told that no budget information could be shared. This makes it impossible to answer key questions about the role of advocacy vs. subsidies, and where future donations are likely to go.
SCI also declined to answer our questions about impact of past work directly, instead suggesting that we search PubMed. We did so, and have also recently reviewed SCI’s updated website, summarizing what we found in this XLS file. Our conclusions:
Deworm The World
We have essentially no information about Deworm the World.
We consider Poverty Action Lab itself to be a model of transparency, posting exhaustive information on studies both completed and in progress. But Deworm the World is on the opposite end of the spectrum, providing no substantive public information as far as we can tell.
We have in the past been tempted to recommend Deworm the World simply on the strength of our respect for Poverty Action Lab. But ultimately, conducting research projects is a different enough challenge from working with governments on large-scale programs that we think that doing so would be the wrong move – both in terms of the incentives it would provide and the good it would accomplish.
*At the time this post was drafted, Poverty Action Lab discussed Deworm the World in various places on its site. However, the links have since broken as Poverty Action Lab redid its website. We have not been able to find any mention of Deworm the World on Poverty Action Lab’s new website.
This is a guest post from Dario Amodei about how he decided what charity to support for his most recent donation. Dario and GiveWell staff had several in-depth conversations as he worked through his decision, so we invited him to share his thought process here. Note that GiveWell has made minor editing suggestions for this post (though Dario determined the final content).
Before I get into the details of my donation decision, I’d like to first share a bit about myself: I’m a graduate student in physics at Princeton, and am interested, very broadly, in what I can do to make the world a better place. I feel that giving away a significant portion of my income is an important part of that, and since 2006 I’ve been donating to organizations that try to improve life in the developing world. I’ve always tried my best to make my donations as effective as possible, but on my own I was never able to give this task as much attention as it deserved. I happened upon GiveWell in 2008 through a link from an economics blog, and to date it’s been the single most useful resource I’ve found in deciding where to donate. Last year I gave $10,000 through GiveWell’s pledge fund, and ultimately decided to allocate all of this money to Village Reach. Holden and Elie have asked me to share the thought process I went through in making my decision, in the hopes that it might be of use to other donors facing a similar choice.
My focus has always been on developing-world health interventions, because I believe these interventions address some of the world’s most urgent needs in a highly tangible way. Six out of 12 of GiveWell’s recommended charities operate in this area, including some health charities I’ve donated to in the past. Reading GiveWell’s reports on these charities, it quickly became clear to me that the “three-star” organizations — Village Reach (VR) and Stop TB — really do stand out above the others. Though I respect and am impressed by the two star organizations, they all seem to have sizable holes in their case for efficacy: for instance, PIH seems to (completely?) lack data on medical outcomes, and the Global Fund seems to have problems with how to use additional funds (William Easterly also seems to have a strongly negative assessment of it in this diavlog ).
Thus, I decided to focus on VR (which aims to improve operational logistics for child vaccinations) and Stop TB (which provides governments with funds for tuberculosis treatment). Choosing between these very compelling charities proved difficult, but I don’t regret the considerable effort I put into my choice — as I tried to constantly remind myself, this choice should involve every bit as much effort as buying a $10,000 item for myself. I considered three relevant factors —
Cost-effectiveness
GiveWell makes explicit cost effectiveness estimates (based in part on those of the Disease Control Priorities report) for both organizations: ~$545 per infant death averted for Village Reach, and ~$150-750 per death averted for Stop TB. These are roughly comparable, but don’t take into account the fact that Stop TB mainly treats adults, while VR mainly treats infants and children. I feel that adults are capable of deeper and more meaningful experiences than are infants, and also deeper connections with other people, so an adult death seems worse to me than an infant death (though both are of course bad). Trying to quantify exactly how much worse is very subjective and can also seem calculating (“how many babies would you kill to save an adult?”), but on a practical level one is forced to make difficult decisions with limited funds, and in my case I’d say that I think an adult death is perhaps 2 or 3 times worse than an infant’s death. Thus, adjusted for my personal values, I’d say that Stop TB is ~2-3 times more cost-effective than VR, though I understand that others may validly disagree with this subjective assessment.
Execution
The second factor, execution, is the one I find most important. By execution I mean all the factors that are assumed to go right in an ideal cost-effectiveness calculation, but could go wrong in practice. I take Murphy’s Law very seriously, and think it’s best to view complex undertakings as going wrong by default, while requiring extremely careful management to go right. This problem is especially severe in charity, where recipients have no direct way of telling donors whether an intervention is working. The situation is worse yet in the developing world, where projects cannot count on the reliable infrastructure and basic social trust we take for granted in the developed world. Given all these problems, what I look for in a charity is a simple and short chain of execution in which relatively few things can go wrong, together with rigorous efforts to close whatever loopholes do exist. As far as I can tell, VR fits these criteria better than any other charity I’ve encountered. Vaccines unquestionably save lives if correctly administered, so it’s generally enough to show that functional vaccines are being correctly delivered and administered. Roughly, the major questions I want answered about a vaccination program are:
(a) are the vaccines actually delivered to health clinics?
(b) do the vaccines remain effective during transport and storage?
(c) once in storage, are the vaccines actually administered, and safely so?
(d) does the program have a clear plan for spending additional money, so that donations actually translate to more vaccines?
(e) are vaccination rates measured to check that the whole chain is working?
I won’t go through the details, which are in GiveWell’s report, but VR makes a systematic effort to address each question. Deliveries are tracked by phone in real-time (e.g. (a)), VR takes an active role in providing power for refrigerators to keep vaccines cold (e.g. (b)), sterilization equipment is provided and stock outs are tracked (which at least suggests successful administration (c)), VR has a clear plan (d) for how to use additional funds, and changes in vaccination rates are measured with controls (e). These steps aren’t perfect – for example, there is apparently no systematic reporting confirming the actual correct administration of vaccines, so step (c) has some room for error — but overall the chain of execution is tighter than any I’ve seen, and the potential holes seem small enough to be manageable.
By contrast, in Stop TB’s case, such a chain (if I could even write it down) would be much longer — Stop TB hands drugs over to governments (involving several layers of administration, differing from country to country) which then must perform all the logistical details VR must perform, plus diagnostics, recurring treatments, and in some cases second-line treatment. There is also the possibility of TB evolving resistance if treatments are not correctly administered. Stop TB’s random inspections, cure rate data, and external auditing seem suggestive of positive results, but my inability to examine in detail a process that I know is quite complex ultimately leaves me very suspicious about efficacy. This isn’t just a matter of Stop TB being a large organization; rather, the problem is that I can’t see the full process of treatment setup and administration, whether applied to one person or a million. Lacking that clear and full view of Stop TB, I have to conclude that VR is the winner on execution.
Incentive effects
Given only VR’s superiority on execution and StopTB’s superiority on cost-effectiveness, I would be about equally inclined to support either, with perhaps a small edge to VR because execution is so critical. However, it’s important to look at the incentive effects of my donation — the money I give out is not just a one-shot intervention, but also a vote on what I want the philanthropic sector to look like in the future. Along these lines, I see three additional advantages to VR, which make it the clear winner in my mind:
In addition to all the arguments listed above, there were a number of other factors which I thought about (some of which were raised in GiveWell’s reports and posts) but ultimately had a hard time getting a handle on and so did not give much weight to. I considered too many factors to list them all, but here are a few examples:
Any one of these effects could theoretically be important enough to outweigh all my arguments for VR, so this list serves as a reminder that there can never be any guarantees of efficacy, let alone optimality. Uncertainty, however, is simply part of life, and all I can do is go with my best guess, so I decided to give to VR.
I hope (though I cannot be sure) that my donation will save the lives of 20 children (which is what the cost-effectiveness numbers work out to). That’s a truly staggering benefit, and honestly it came at very little cost to myself: I don’t much miss the new car I didn’t buy, and I’ll gladly make the same sacrifice next year in order to donate again. What did feel very emotionally taxing was reading (and in most cases, agreeing with) all the negative analysis of charities at GiveWell and elsewhere. I found it difficult to evaluate everything in a critical fashion while still holding on to the compassion and optimism that originally inspired me to donate. It’s tough to find the right balance between caring and hard-nosed realism, but it is possible, and it is, as far as I know, the only way to truly change the world.
Among the groups in our “meta-philanthropy” space, one of the big questions is how to create more “high-impact” (also called “blue-chip”) charities: the rare groups that can reliably, demonstrably translate donations into improved lives.
The rough consensus seems to be that we need to fund and support “high-performance organizations”: groups that have “some, but not all” of the qualities of blue-chip charities. The idea is that charities start with a few good qualities and slowly grow into blue-chippers.
(See, for example, parts one and two of Tactical Philanthropy’s discussion from last summer, which argues that “the best thing to do is not study how to [achieve impact] and then set out on that exact path with the exact tools needed according to your theory, but instead to build the most robust [organization] possible.” Also see the most recent draft of the Social Risk Assessment Protocol, which effectively gives twice the weight to a charity’s evaluation and adjustment practices that it gives to the charity’s choice of programs.)
I’ve been rethinking this idea. When I look at the three charities that I consider to be most “blue-chip” (our three highest-ranked charities), I don’t see a path of a “strong organization that eventually figured out what to do and whether it worked. ” Rather, I see organizations that stayed as small as possible – or didn’t exist – until they had strong evidence of impact for their basic approach. They built their choice of programming into their DNA, as much as they could, from day one.
The reigning consensus seems to treat “evidence of impact” as a late (or at least potentially late) step in the development of a nonprofit, but in fact it has been the first step for the strongest nonprofits I know of.
This makes intuitive sense to me as well.
The truth is that if an organization wants to become “high-impact,” there are already proven approaches for it to choose from; if it wants to investigate an approach that isn’t yet proven, it can (like VillageReach) stay at minimal size and essentially act as a “research project.” For an organization that has chosen to do neither of these things, and has already “scaled up” its program and built up a staff/organization (no matter how well run) … it may be too late.
GiveWell focuses on finding blue-chip charities, not on creating them. But for those looking to do the latter, I submit that it may be less effective to start with “high-performance organizations” than to start from scratch.