The GiveWell Blog

GiveWell’s annual self-evaluation and plan: A big picture change in priorities

(updated on: ) | by Holden

As we did last year, we’re going to be posting our annual self-evaluation and plan as a series of blog posts.

Our self-evaluation starts with a set of critical questions about GiveWell, intended to examine how well we’re accomplishing our high-level goals. For the most part, our high-level goals are the same as they were last year, and thus our questions for GiveWell are the same as the questions we laid out last year. However, there is one major change in how we see the goals of GiveWell: we no longer assign high importance to the number of causes covered / number of options we provide to donors. Instead, we just want to focus on finding outstanding charities that donors can be confident in, and closing those charities’ funding gaps to the extent that we can.

We see charity evaluation groups as falling into the following possible categories:

  1. Groups that rate as many charities as possible. Donors come to them already having a particular charity in mind to give to, and search for that charity.
  2. Groups that suggest charities for as many causes as possible. Donors come to them knowing what sort of cause they want to support (U.S. education, global health, etc.) but not which charity, and get a recommendation.
  3. Groups that simply focus on finding outstanding charities. Donors come to them looking for outstanding giving opportunities (they are often issue-agnostic).

We started GiveWell as issue-agnostic donors looking for the best giving opportunities we could find, and we have always primarily been interested in #3. We’ve never had a serious interest in #1 above – distinguishing between “worst,” “bad,” “good,” and “better” is too much of a distraction when what we care about is “best.” But for most of our history, we’ve seen ourselves as possibly being on a trajectory toward becoming #2 in addition to #3. We’ve talked about covering a broad array of causes to interest as many donors as possible, thus increasing our influence and visibility. Many of those who have criticized us have focused on the small number of causes we’ve covered, and expressed the hope that we will eventually cover many more.

A couple of things have made us rethink this goal:

1. We see it as urgent, and difficult, to find more “gold medal” charities than we have so far.

When the 2010 holiday season was approaching, and we started thinking about our strategy for outreach, we realized that we only had one charity we could personally feel really good about aggressively raising money for. While we think all our recommended charities stand well above other donors’ options, it was only VillageReach for which I felt I could sit down with someone face-to-face about and say, “Give as much as you can to this one.”

Why? Because not only does VillageReach have outstanding evidence of effectiveness; it has outstanding bang-for-the-buck (in absolute terms, not just relative terms) and most importantly, it has a concrete plan for additional funding.

We personally don’t like raising money for groups that we can’t say all these things about. We’re happy to provide our recommendations for someone looking for the best microfinance charity, but we can’t honestly say that we think dollars given to it are accomplishing as much good as possible.

VillageReach had a “stretch” fundraising target for the year of $1.5 million. That gap has now been more than closed. VillageReach needs a total of $4.4 million for its multi-year project, so we are happy to leave it as our top-rated charity for now, but if our growth continues, the entire expansion could be funded by the end of the year. Thus, we are starting to get dangerously close to the point where the number of dollars we influence exceeds the number of dollars we know how to allocate very effectively.

We see this as an urgent situation. It would be a major problem for GiveWell if we essentially had more demand for our research (i.e., donors interested in following our recommendations) than supply (i.e., charities able to absorb this funding effectively). Thus, one of our top priorities for 2011 is finding more “gold medal” charities that we can give a wholehearted recommendation. We feel it would be a major distraction – and mistake – to try to find the “best of the bunch” within causes where there are no groups that really shine by our criteria.

2. The benefits of covering extra causes don’t seem very large.

The argument we’d always used for covering more causes was that covering more causes means appealing to more donors. In theory, this is true, but for the specific audience we seem to be attracting, the value of breadth seems surprisingly low.

The following charts show the percentage of our 2010 “money moved” (donations given to our recommended charities as a result of our research, which we track in a variety of ways) to different categories of charity. A future post will go into more detail on how we track “money moved” and what the figures were for 2010.


Note that VillageReach is our top-rated charity; Against Malaria Foundation is our only recommended charity that allows donors to get tax deductions in a variety of countries outside the U.S. (Canada, Australia and more); Nurse-Family Partnership (NFP) is our top-rated U.S. charity and was featured in a Nicholas Kristof column (we count donors coming from that column as “money moved” because Mr. Kristof has informed us that he included NFP in the column on our recommendation).

Bottom line – it looks like the overwhelming majority of our donors follow our recommendations regarding the promising cause and the most promising organization.

Even our top-rated U.S. charity didn’t attract nearly as much funding from our donors as our global health charities did. Meanwhile, the work we’ve put into accommodating donors interested in narrower causes – education and microfinance – has barely registered in terms of increased “money moved.”

And education and microfinance are still pretty broad, popular causes. I’d expect a “cancer research” or “global warming” recommendation to be pretty comparable to education and microfinance in terms of new money moved; I’d expect a narrower cause like “homelessness in New York City” to bring much less.

This doesn’t mean that all donors are issue-agnostic or global health fans. It means that our audience is. And we need to focus on serving our audience as well as possible.

Implications

We aren’t dropping the idea of “offering recommendations within lots of causes” entirely. We aren’t completely sure why our audience is the way it is, and recognize that over time the situation may change. Thus,

  • We plan to maintain/update our existing research on all causes, including the causes above plus disaster relief (the newest addition).
  • We plan to investigate more causes for the purpose of finding more outstanding charities. We will be writing up as much as we can of what we learn in these investigations, which may (along with our do-it-yourself evaluation guide) be helpful to donors interested in particular causes.
  • We may provide donors a way to “lobby us” to cover more causes. I’ve thought about the idea of offering donors the chance to commit to give $X to our top-rated charity in a given cause, conditional on our covering that cause; if we got enough commitments for a given cause we would cover it.

However, we no longer consider “number of causes covered” a relevant metric for GiveWell and will be replacing it with “room for money moved,” i.e., the total amount of room for more funding of our very top-rated charities.

Some may be disappointed by this decision, seeing it as a retreat from the opportunity to be a resource for as many people as possible. But we’re not sure how much sense it really makes for an operation like ours to maximize its breadth. Giving decisions are deeply personal, and the kind of work we do is as well. We’ve mostly followed our personal values in making recommendations, and we’re attracting an audience that seems comfortable with these values.

This doesn’t mean that there’s no place for a GiveWell-type operation that focuses on (for example) U.S. causes; but perhaps we aren’t the people to run that operation. Perhaps it takes someone who truly believes that U.S. charities represent the best chance to help people (we do not) to do compelling research and attract the right audience for that cause. We’d be more than happy to see such a group spring up. For our part, we’re happy to remain a niche operation for a niche audience, as long as the niche is big enough. And it appears to be growing.

Global Fund: Best failure disclosure we’ve seen yet?

(updated on: ) | by Holden

Over the last year or so, The Global Fund has disclosed instances of apparent fraud in Zambia, Mali, and Mauritania among other countries.

This week, The Associated Press reported on these disclosures with the hostile headline: “Fraud plagues global health fund.” The Global Fund has been defended with the valid observation that the total fraud found accounts for a tiny percentage of its overall grants. But we’d like to do more than defend the Global Fund; we’d like to praise it for these disclosures.

The idea that disclosing failures should be rewarded has gained steam in our corner of the sector, particularly with the new AdmittingFailure.com site (see coverage by Good Intentions Are Not Enough and Tactical Philanthropy). We agree with the idea (previous posts on the topic here and here). But scanning the compilation at AdmittingFailure.com, we don’t see any admissions of failure that are as concrete, specific, and risky as the fraud disclosures the Global Fund has been making with some regularity.

To us, the key points regarding the Global Fund (aside from the fact that the percentage of funding reported as misused is small) are that

  • Disclosing these issues is a choice made by the Global Fund. In our investigations of large charities, we’ve seen no evidence that donors are auditing them in a way that would force disclosure of lost funds. We’ve also seen no evidence that these organizations have any way of even knowing when funds have been misused. Take UNICEF for an example: if UNICEF lost millions of dollars to fraud, it isn’t clear to us how (or whether) anyone would find out about it.
  • Other large charities could easily be seeing as much fraud, or more. We believe it is difficult to get anything done overseas without significant local help. Though it can be hard to tell, we believe that large organizations usually work with governments and with smaller community-based organizations, giving both opportunities to misappropriate funds. Even when they are officially executing their own programs, there’s a large number of degrees of separation between donors in the U.S. and local/locally connected people assigned to execute on the ground. We don’t see any reason to expect a systematically higher level of honesty from these people than from the people the Global Fund has worked with.

Bottom line – we feel that any large bureaucratic organization, particularly one that does a lot of grantmaking, could be losing a lot of money to fraud; what’s unusual about The Global Fund is that it is actively searching for these cases, disclosing them publicly, and then discussing (also publicly) how they can be addressed (PDF).

The Global Fund is not one of our top-rated charities; we think there are groups with even stronger reporting, and/or less complexity and bureaucracy to their activities, that we prefer for maximum impact. However, this incident reinforces our belief that The Global Fund has outstanding transparency compared to similar organizations. We think its disclosures of fraud deserve a place on AdmittingFailure.com and praise, not hostile headlines.

Free legal services – helpful or harmful?

(updated on: ) | by Elie

The Harvard Legal Aid Bureau provides free legal services, via volunteer Harvard Law students, to “low-income people in civil (non-criminal) matters in order to ensure equal access to justice and to remove legal barriers to economic opportunity.”

It’s about as intuitive an intervention as any. Good legal services are expensive; the services provided by Harvard Law students are probably good; so it seems they would be beneficial to the low-income people receiving them. But a recent (unpublished) study calls this into question, implying that the benefits of the program (at least the service studied) are small and that there are possible harms as well.

Before getting into the details of the study, it’s important to credit the Harvard Legal Aid Bureau for participating in this study. Many organizations would likely shy away from a study that could find negative results. The Bureau deserves praise for participating in the study, and our opinion of them is now higher for their participation.

The study uses a randomized controlled trial methodology to evaluate the impact of offering legal services to clients seeking unemployment benefits. The paper finds that:

  • An offer of services had no statistically significant effect on the likelihood of an individual’s receiving government benefits. 76% of those offered services (the treatment group) by the Harvard group received benefits; 72% of those who were not offered services (the control group) received benefits.
  • An offer of services had a statistically significant effect on the delay until an individual received benefits. The treatment group received services more than 13 days later than the control group. The paper argues that those applying for unemployment benefits often have an immediate need for cash, and this delay is a significant negative impact of the program.

What explains the results? The authors offer a number of suggestions.

  • The authors note that they used an intent-to-treat methodology in this paper. That is, they assessed the impact of the offer of services, not services, themselves. Ultimately, 90%+ of the treatment group received services while only 49% of the control group did. The fact that nearly half the control group received services may explain the lack of a statistically significant effect of offering services. We believe that using an intent-to-treat approach was the correct approach here, especially given the question most important to us at GiveWell: what is the marginal impact of a particular organization? In the absence of the Harvard Legal Aid Bureau, it seems that approximately half of the clients they serve would be served elsewhere.
  • The Harvard Legal Aid Bureau is staffed by law students (under the supervision of a practicing attorney). The authors consider the possibility that because of their limited experience, the students are less effective than practicing attorneys would be. Nevertheless, the authors conclude — largely it seems based on their observations of the students and experience with similar organizations — that the students are likely competent and this is unlikely a cause of the lack of impact.
  • The authors suggest that the additional delay experienced by students’ clients may be due to the students’ schedules and the fact that they only commit part-time to the legal aid bureau.

It’s only one study and it shouldn’t be taken too far – it addresses the impact of one service offered by one particular legal aid organization, and doesn’t show that free legal services for low-income people are unnecessary. But it does hint that

  • From a donor perspective, legal aid charities may not have room for more funding. (Here it looks like the supply was enough that the existence of a particular group, Harvard’s, couldn’t be connected to strong impact.)
  • A sensible-seeming program can have unintended consequences. In this case, matching low-income people with law students at a prestigious institution might also mean matching them with people who don’t have enough time to address their issues promptly.

Disaster relief report published; Doctors Without Borders, Partners In Health, and Direct Relief International stand out

(updated on: ) | by Holden

Our new report grades major disaster relief organizations on their transparency and accountability to donors. It provides detailed reports on each charity, a summary table with our conclusions, and the full details of our process.

We conclude that Doctors Without Borders, Partners in Health and Direct Relief International stand out for the clarity with which they discuss their activities and expenditures.

Over the coming months, we’ll be adding to this report by examining information from the response to the 2004 Asian tsunami. We’ll also be providing some more discussion of non-transparency-related factors behind the donation decision. For example, Direct Relief International and Doctors Without Borders are very different in their focus – the former focuses on distributing supplies while the latter largely facilitates and supplies medical treatment. Personally, I find the latter to be a more appealing vehicle for addressing pressing and challenging needs in a relief situation. Other donors may prefer to support organizations with broader mandates, which are more likely to play direct roles in (for example) providing shelter and assisting with longer-term reconstruction.

That said, we feel that what we have now is a substantial improvement over the information and analysis previously available to donors. It consolidates the clearest and most detailed information provided by each organization, and it can help donors take a first step toward creating incentives for organizations to raise money by being more transparent (and not just more pushy in fundraising) than their competitors.

While we haven’t assessed the quality of individual organizations’ relief efforts, we have accompanied our report with a general overview of what the overall relief effort has and hasn’t accomplished in the year since the disaster, and for how much.

GiveWell’s report on major disaster relief organizations

What is the situation in Haiti a year after the earthquake? What have and haven’t charities accomplished so far?

(updated on: ) | by Holden

Yesterday we discussed how much has been raised and spent for Haiti relief. Today we’ll summarize what we know about how the relief effort has progressed over the last year.

Our detailed and sourced account of the relief effort as a whole will be available by the end of today, and linked here when it is. Update: this page is now available. (Our take on individual organizations will be published tomorrow.) For now, the big picture as we see it is that the relief effort has reached a lot of people with some basic necessities, but that conditions in the camps are still extremely poor, and that there is a dire need to halt the ongoing outbreak of cholera and clear more of the rubble.

  • The relief effort provided immediate shelter assistance, mostly in the form of tarps, within three months of the earthquake, although there has been some criticism that this was slower than it needed to be due to coordination issues. There has also been criticism of the emphasis on tarps as opposed to tents.
  • Conditions in settlement camps, while varied, have generally been extremely poor. One study involving visits to over 100 camps concluded that

    seven months following the earth-quake, 40 percent of … camps do not have access to water, and 30 percent do not have toilets of any kind. An estimated 10 percent of families have a tent; the rest sleep under tarps or even bed sheets. In the midst of the hurricane season with torrential rains and heavy winds a regular occurrence, many tents are ripped beyond repair. Only a fifth of camps have education, health care, or psycho-social facilities on site.

  • Construction of transitional shelters (higher-quality living spaces compared to camps) has been far slower than hoped. 9 months after the earthquake, only about 60,000 people were living in such shelters (out of likely over 1 million people left homeless by the earthquake). The number of transitional shelters has reportedly tripled since then, so things may be improving on this front. Rubble and confusion over land rights have been major obstacles to transitional shelter construction.
  • Water and sanitation efforts have been hampered by the difficulty of operating in a crowded urban area, and have generally been poor, especially in terms of sensitivity to privacy. A massive outbreak of cholera began in October and has led to over 3000 deaths and 171,000 infections nationwide, and is ongoing.
  • A large number of people have been reached with medical assistance and food aid, and we have not seen major criticisms of the relief effort on these fronts. We have also seen no assessments of the quality of medical care or of medical outcomes after the earthquake (i.e., deaths/complications not directly related to the earthquake itself or the cholera outbreak).
  • Rubble removal has been a major problem, and at least 80% (possibly much more) of rubble remains un-managed. Property rights and coordination issues have been obstacles on this front; the difficulty of navigating narrow roads has been an issue as well.

Overall, we’d say that the progress of relief has been disappointing.

One of the questions we’ve been thinking about is whether relief in a situation like this is over- or under-funded relative to everyday aid. I see a few possible interpretations of the disappointing relief effort in Haiti:

  1. Relief organizations aren’t spending money fast enough – they are selfishly/irrationally holding money for later projects that they should be spending now. If they would spend more, the above problems would be alleviated.
  2. Relief organizations are wisely conserving their funds for necessary later rebuilding efforts. If donors gave more generously, relief organizations would be spending more now, and still have enough left over for later rebuilding.
  3. Relief organizations are wisely withholding funds because money isn’t the bottleneck to better outcomes. The logistical and political problems can’t be solved simply by spending more money, and any spending above current levels could be wasteful and even harmful.

Although #1 seems to be the most common narrative in the media, I find it the hardest to believe. All of the public pressure seems to be on nonprofits to spend faster and get quicker, more tangible results. Spending money now seems to be the best move from a public relations standpoint; if it were also the best move from an outcomes standpoint, I don’t see what motivation relief organizations would have for doing otherwise.

#2 seems possible. We have acknowledged that rebuilding Haiti could take all the money that’s been given and more.

However, given the direness and urgency of the current needs – particularly the cholera outbreak and the rubble situation – it seems to me that any effective investment in getting better outcomes now ought to more than pay off later. (Haiti can’t be rebuilt without clearing the rubble or stopping the cholera outbreak; the sooner these are done, the better.) Because of this, and in view of the large amounts given/spent in the context of Haiti’s economy, I lean toward explanation #3: Haiti earthquake relief doesn’t have immediate room for more funding (though this would not preclude having significant needs for more long-term rebuilding funds).

How much money has been given and spent for Haiti earthquake relief? Putting the numbers in perspective

(updated on: ) | by Holden

We’ll soon be releasing an assessment of disaster relief organizations. Though we haven’t found it practical (yet) to evaluate the effectiveness of their work, we’d like to take a broad look at what the relief effort as a whole has and hasn’t accomplished, and for how much money.

This post focuses on the latter: how much funding has come in for disaster relief? Enough to compensate for all the damage that was caused? Enough to make a major dent in people’s suffering? Or not enough to do much of anything?

In brief, it looks to us as though

  • About $5.2 billion has been raised or pledged; about $1.6 billion has been spent.
  • Big-name charities set multi-year fundraising targets in February and had come close to them by July.
  • The amount raised/pledged is equal to about 80% of Haiti’s GDP, and could theoretically provide each person affected by the earthquake with 2.6 years’ worth of a Haiti-average income, or could provide each person left homeless with 5-8 years’ worth of a Haiti-average income.
  • The amount spent is equal to about 25% of Haiti’s GDP, and could theoretically provide each person affected by the earthquake with about 9 months’ worth of a Haiti-average income, or could provide each person left homeless with 1.6-2.4 years’ worth of a Haiti-average income.
  • The total value of the damage could easily be greater than the total amount raised/pledged.

Considering the extent of the damage and logistical challenges, we wouldn’t necessarily expect all of the damage to be repaired, even 5 years from now. Therefore, we shouldn’t be surprised to see continued suffering and challenges in Haiti, and we also shouldn’t be surprised or upset that organizations are holding much of the money they’ve raised, for later rebuilding efforts. However, given the large amount available (and the large amount spent) in the context of Haiti’s economy, we should expect to see major and tangible benefits coming from the relief effort.

Details of the numbers above follow.

How much has been given?

The most comprehensive-seeming source we’ve found for disaster relief funding is the OCHA list of all humanitarian pledges, commitments & contributions in 2010 (XLS) found in the collection of 2010 Haiti-related documents on Reliefweb (“OCHA” for the remainder of this post). OCHA gives a total of about $3.5 billion committed plus another $1 billion in uncommitted pledges.

However, this isn’t the whole story – it seems to us that the OCHA data is missing a substantial amount. We compared the OCHA numbers to Chronicle of Philanthropy figures from a July 2010 survey. When there was a charity that we were confident we could identify in both tallies, we put the Chronicle figure for that charity side-by-side with the OCHA figure (see the “OCHA vs Chronicle – diffs” sheet in our analysis (XLS)). Even though the OCHA figures are as of January 2011 while the Chronicle figures are as of July 2010, the Chronicle figure exceeds the OCHA figure for the vast majority of matched charities, sometimes by a very large amount. Just looking at the charities that we can confidently match up between the two implies over $700 million in uncounted funding in the OCHA tally.

These matched charities tend to be the big ones, and collectively account for ~$1.37 billion out of the ~$1.65 billion total for all the figures in the Chronicle tally. (Note that we are using “worldwide funds raised” figures when available; using only U.S. figures would generate a total of $1.3 billion, consistent with the Chronicle’s own figure for the total). Furthermore, the funding that’s come in between July and now doesn’t seem significant – the latest Chronicle total for U.S. giving is $1.4 billion, compared to $1.3 billion in the July report.

So we’d say that the total amount of money given or pledged appears to be at least $5.2 billion (~$3.5 billion given according to the OCHA report; ~$1 billion pledged from OCHA report; ~$700 million in uncounted funding that we identified in the Chronicle tally). More thoroughly counting all the “missed funding” for particular nonprofits could raise this total, though probably not by very much.

The recent Chronicle update also states that about 38% of the funds raised have been spent. This applies to the charities in the Chronicle tally, not to all the agents included in the OCHA tally, but we can estimate the amount spent by multiplying our estimate of the total amount committed (not pledged) by 38%, to get $1.6 billion spent.

Have charities raised as much as they expected/hoped to?

Here’s a comparison of how much a few large charities said they needed in February 2010 (from a Chronicle survey) vs. how much they had raised as of July 2010 (from another Chronicle survey).

Charity Requested as of 2/21/10 Received as of 7/9/10
Partners in Health At least $100 million $85 million
Oxfam At least $100 million $90 million
Save the Children $85 million to $115 million $71.4 million
UNICEF $128 million $73.1 million (US only)
World Vision At least $100 million $192 million

Note that the “Requested” figures above are multi-year appeals.

We don’t want to read too much into this data. The announced “funding needs” may have been formulated more based on what was achievable (i.e., trying to set a “stretch target”) than on what expenses were expected – we don’t know. But the basic picture seems to be that major charities raised close to what they were seeking in the few months after the disaster.

How much has been spent/raised in the context of Haiti’s economy and the damage it suffered?

According to the CIA World Factbook, Haiti had an estimated 2009 GDP of $6.56 billion and a population of about 9.6 million, which implies an average annual income (in U.S. dollars, not adjusted for purchasing power parity) of $680.

$5.2 billion (the amount raised/pledged) is equal to about 80% of Haiti’s total yearly economic output; $1.6 billion (the amount spent) is equal to about 25%.

Using an estimate that 30% of the population was affected by the earthquake (see below), $5.2 billion (the amount raised/pledged) would theoretically be enough to give each affected person about $1700, or 2.7 years’ worth of Haiti-average income (though people affected by the earthquake were largely in urban areas and thus probably had above-average incomes for Haiti). $1.6 billion (the amount spent) would be enough for $533, or about 9 months’ worth of Haiti-average income.

Using an estimate that 1-1.5 million people were left homeless by the earthquake (see below), $5.2 billion would theoretically be enough to give each person left homeless about $3500-$5200, or about 5.1-7.7 years’ worth of Haiti-average income. $1.6 billion (the amount spent) would theoretically be enough to give each person left homeless $1067-$1600, or 1.6-2.4 years’ worth of Haiti-average income.

That wouldn’t necessarily be enough to compensate for all the damage, especially in a logistically challenging situation. One study (PDF) estimated that the earthquake caused $7.2 billion-$13.9 billion worth of damages, which is 1.5-3x the amount of aid given and 1.1-2x the value of Haiti’s annual GDP. We find this study highly unreliable (it is attempting to predict the value of the damage based on a linear regression using a very small number of predictive variables such as the number of people killed, land area, population, GDP and disaster type – see page 6), but we do find it plausible that the total losses for a person left homeless could exceed 8 years’ worth of their income, and thus we find it plausible that the amount of aid raised/pledged is not equal to the value of the damage caused.

A note on figures for “people affected” and “people left homeless”

Unfortunately we haven’t yet been able to find official government or U.N. estimates for these figures, accompanied by details of the estimation procedure; the best we’ve found is various news stories citing the government or nonprofit agencies. The number we’ve seen for “total people affected” is 3 million (citing the International Federation of the Red Cross) and the numbers we’ve seen for “people left homeless” are 1 million (citing the government) to 1.5 million (we’ve seen this number in several places including the Chronicle of Philanthropy but haven’t seen a source for it). This would imply that the earthquake affected about 30% of Haiti’s total population and left 10-15% of the total population homeless (using the CIA World Factbook population estimate of 9.6 million).