The GiveWell Blog

Global Fund: Best failure disclosure we’ve seen yet?

(updated on: ) | by Holden

Over the last year or so, The Global Fund has disclosed instances of apparent fraud in Zambia, Mali, and Mauritania among other countries.

This week, The Associated Press reported on these disclosures with the hostile headline: “Fraud plagues global health fund.” The Global Fund has been defended with the valid observation that the total fraud found accounts for a tiny percentage of its overall grants. But we’d like to do more than defend the Global Fund; we’d like to praise it for these disclosures.

The idea that disclosing failures should be rewarded has gained steam in our corner of the sector, particularly with the new AdmittingFailure.com site (see coverage by Good Intentions Are Not Enough and Tactical Philanthropy). We agree with the idea (previous posts on the topic here and here). But scanning the compilation at AdmittingFailure.com, we don’t see any admissions of failure that are as concrete, specific, and risky as the fraud disclosures the Global Fund has been making with some regularity.

To us, the key points regarding the Global Fund (aside from the fact that the percentage of funding reported as misused is small) are that

  • Disclosing these issues is a choice made by the Global Fund. In our investigations of large charities, we’ve seen no evidence that donors are auditing them in a way that would force disclosure of lost funds. We’ve also seen no evidence that these organizations have any way of even knowing when funds have been misused. Take UNICEF for an example: if UNICEF lost millions of dollars to fraud, it isn’t clear to us how (or whether) anyone would find out about it.
  • Other large charities could easily be seeing as much fraud, or more. We believe it is difficult to get anything done overseas without significant local help. Though it can be hard to tell, we believe that large organizations usually work with governments and with smaller community-based organizations, giving both opportunities to misappropriate funds. Even when they are officially executing their own programs, there’s a large number of degrees of separation between donors in the U.S. and local/locally connected people assigned to execute on the ground. We don’t see any reason to expect a systematically higher level of honesty from these people than from the people the Global Fund has worked with.

Bottom line – we feel that any large bureaucratic organization, particularly one that does a lot of grantmaking, could be losing a lot of money to fraud; what’s unusual about The Global Fund is that it is actively searching for these cases, disclosing them publicly, and then discussing (also publicly) how they can be addressed (PDF).

The Global Fund is not one of our top-rated charities; we think there are groups with even stronger reporting, and/or less complexity and bureaucracy to their activities, that we prefer for maximum impact. However, this incident reinforces our belief that The Global Fund has outstanding transparency compared to similar organizations. We think its disclosures of fraud deserve a place on AdmittingFailure.com and praise, not hostile headlines.

Free legal services – helpful or harmful?

(updated on: ) | by Elie

The Harvard Legal Aid Bureau provides free legal services, via volunteer Harvard Law students, to “low-income people in civil (non-criminal) matters in order to ensure equal access to justice and to remove legal barriers to economic opportunity.”

It’s about as intuitive an intervention as any. Good legal services are expensive; the services provided by Harvard Law students are probably good; so it seems they would be beneficial to the low-income people receiving them. But a recent (unpublished) study calls this into question, implying that the benefits of the program (at least the service studied) are small and that there are possible harms as well.

Before getting into the details of the study, it’s important to credit the Harvard Legal Aid Bureau for participating in this study. Many organizations would likely shy away from a study that could find negative results. The Bureau deserves praise for participating in the study, and our opinion of them is now higher for their participation.

The study uses a randomized controlled trial methodology to evaluate the impact of offering legal services to clients seeking unemployment benefits. The paper finds that:

  • An offer of services had no statistically significant effect on the likelihood of an individual’s receiving government benefits. 76% of those offered services (the treatment group) by the Harvard group received benefits; 72% of those who were not offered services (the control group) received benefits.
  • An offer of services had a statistically significant effect on the delay until an individual received benefits. The treatment group received services more than 13 days later than the control group. The paper argues that those applying for unemployment benefits often have an immediate need for cash, and this delay is a significant negative impact of the program.

What explains the results? The authors offer a number of suggestions.

  • The authors note that they used an intent-to-treat methodology in this paper. That is, they assessed the impact of the offer of services, not services, themselves. Ultimately, 90%+ of the treatment group received services while only 49% of the control group did. The fact that nearly half the control group received services may explain the lack of a statistically significant effect of offering services. We believe that using an intent-to-treat approach was the correct approach here, especially given the question most important to us at GiveWell: what is the marginal impact of a particular organization? In the absence of the Harvard Legal Aid Bureau, it seems that approximately half of the clients they serve would be served elsewhere.
  • The Harvard Legal Aid Bureau is staffed by law students (under the supervision of a practicing attorney). The authors consider the possibility that because of their limited experience, the students are less effective than practicing attorneys would be. Nevertheless, the authors conclude — largely it seems based on their observations of the students and experience with similar organizations — that the students are likely competent and this is unlikely a cause of the lack of impact.
  • The authors suggest that the additional delay experienced by students’ clients may be due to the students’ schedules and the fact that they only commit part-time to the legal aid bureau.

It’s only one study and it shouldn’t be taken too far – it addresses the impact of one service offered by one particular legal aid organization, and doesn’t show that free legal services for low-income people are unnecessary. But it does hint that

  • From a donor perspective, legal aid charities may not have room for more funding. (Here it looks like the supply was enough that the existence of a particular group, Harvard’s, couldn’t be connected to strong impact.)
  • A sensible-seeming program can have unintended consequences. In this case, matching low-income people with law students at a prestigious institution might also mean matching them with people who don’t have enough time to address their issues promptly.

Disaster relief report published; Doctors Without Borders, Partners In Health, and Direct Relief International stand out

(updated on: ) | by Holden

Our new report grades major disaster relief organizations on their transparency and accountability to donors. It provides detailed reports on each charity, a summary table with our conclusions, and the full details of our process.

We conclude that Doctors Without Borders, Partners in Health and Direct Relief International stand out for the clarity with which they discuss their activities and expenditures.

Over the coming months, we’ll be adding to this report by examining information from the response to the 2004 Asian tsunami. We’ll also be providing some more discussion of non-transparency-related factors behind the donation decision. For example, Direct Relief International and Doctors Without Borders are very different in their focus – the former focuses on distributing supplies while the latter largely facilitates and supplies medical treatment. Personally, I find the latter to be a more appealing vehicle for addressing pressing and challenging needs in a relief situation. Other donors may prefer to support organizations with broader mandates, which are more likely to play direct roles in (for example) providing shelter and assisting with longer-term reconstruction.

That said, we feel that what we have now is a substantial improvement over the information and analysis previously available to donors. It consolidates the clearest and most detailed information provided by each organization, and it can help donors take a first step toward creating incentives for organizations to raise money by being more transparent (and not just more pushy in fundraising) than their competitors.

While we haven’t assessed the quality of individual organizations’ relief efforts, we have accompanied our report with a general overview of what the overall relief effort has and hasn’t accomplished in the year since the disaster, and for how much.

GiveWell’s report on major disaster relief organizations

What is the situation in Haiti a year after the earthquake? What have and haven’t charities accomplished so far?

(updated on: ) | by Holden

Yesterday we discussed how much has been raised and spent for Haiti relief. Today we’ll summarize what we know about how the relief effort has progressed over the last year.

Our detailed and sourced account of the relief effort as a whole will be available by the end of today, and linked here when it is. Update: this page is now available. (Our take on individual organizations will be published tomorrow.) For now, the big picture as we see it is that the relief effort has reached a lot of people with some basic necessities, but that conditions in the camps are still extremely poor, and that there is a dire need to halt the ongoing outbreak of cholera and clear more of the rubble.

  • The relief effort provided immediate shelter assistance, mostly in the form of tarps, within three months of the earthquake, although there has been some criticism that this was slower than it needed to be due to coordination issues. There has also been criticism of the emphasis on tarps as opposed to tents.
  • Conditions in settlement camps, while varied, have generally been extremely poor. One study involving visits to over 100 camps concluded that

    seven months following the earth-quake, 40 percent of … camps do not have access to water, and 30 percent do not have toilets of any kind. An estimated 10 percent of families have a tent; the rest sleep under tarps or even bed sheets. In the midst of the hurricane season with torrential rains and heavy winds a regular occurrence, many tents are ripped beyond repair. Only a fifth of camps have education, health care, or psycho-social facilities on site.

  • Construction of transitional shelters (higher-quality living spaces compared to camps) has been far slower than hoped. 9 months after the earthquake, only about 60,000 people were living in such shelters (out of likely over 1 million people left homeless by the earthquake). The number of transitional shelters has reportedly tripled since then, so things may be improving on this front. Rubble and confusion over land rights have been major obstacles to transitional shelter construction.
  • Water and sanitation efforts have been hampered by the difficulty of operating in a crowded urban area, and have generally been poor, especially in terms of sensitivity to privacy. A massive outbreak of cholera began in October and has led to over 3000 deaths and 171,000 infections nationwide, and is ongoing.
  • A large number of people have been reached with medical assistance and food aid, and we have not seen major criticisms of the relief effort on these fronts. We have also seen no assessments of the quality of medical care or of medical outcomes after the earthquake (i.e., deaths/complications not directly related to the earthquake itself or the cholera outbreak).
  • Rubble removal has been a major problem, and at least 80% (possibly much more) of rubble remains un-managed. Property rights and coordination issues have been obstacles on this front; the difficulty of navigating narrow roads has been an issue as well.

Overall, we’d say that the progress of relief has been disappointing.

One of the questions we’ve been thinking about is whether relief in a situation like this is over- or under-funded relative to everyday aid. I see a few possible interpretations of the disappointing relief effort in Haiti:

  1. Relief organizations aren’t spending money fast enough – they are selfishly/irrationally holding money for later projects that they should be spending now. If they would spend more, the above problems would be alleviated.
  2. Relief organizations are wisely conserving their funds for necessary later rebuilding efforts. If donors gave more generously, relief organizations would be spending more now, and still have enough left over for later rebuilding.
  3. Relief organizations are wisely withholding funds because money isn’t the bottleneck to better outcomes. The logistical and political problems can’t be solved simply by spending more money, and any spending above current levels could be wasteful and even harmful.

Although #1 seems to be the most common narrative in the media, I find it the hardest to believe. All of the public pressure seems to be on nonprofits to spend faster and get quicker, more tangible results. Spending money now seems to be the best move from a public relations standpoint; if it were also the best move from an outcomes standpoint, I don’t see what motivation relief organizations would have for doing otherwise.

#2 seems possible. We have acknowledged that rebuilding Haiti could take all the money that’s been given and more.

However, given the direness and urgency of the current needs – particularly the cholera outbreak and the rubble situation – it seems to me that any effective investment in getting better outcomes now ought to more than pay off later. (Haiti can’t be rebuilt without clearing the rubble or stopping the cholera outbreak; the sooner these are done, the better.) Because of this, and in view of the large amounts given/spent in the context of Haiti’s economy, I lean toward explanation #3: Haiti earthquake relief doesn’t have immediate room for more funding (though this would not preclude having significant needs for more long-term rebuilding funds).

How much money has been given and spent for Haiti earthquake relief? Putting the numbers in perspective

(updated on: ) | by Holden

We’ll soon be releasing an assessment of disaster relief organizations. Though we haven’t found it practical (yet) to evaluate the effectiveness of their work, we’d like to take a broad look at what the relief effort as a whole has and hasn’t accomplished, and for how much money.

This post focuses on the latter: how much funding has come in for disaster relief? Enough to compensate for all the damage that was caused? Enough to make a major dent in people’s suffering? Or not enough to do much of anything?

In brief, it looks to us as though

  • About $5.2 billion has been raised or pledged; about $1.6 billion has been spent.
  • Big-name charities set multi-year fundraising targets in February and had come close to them by July.
  • The amount raised/pledged is equal to about 80% of Haiti’s GDP, and could theoretically provide each person affected by the earthquake with 2.6 years’ worth of a Haiti-average income, or could provide each person left homeless with 5-8 years’ worth of a Haiti-average income.
  • The amount spent is equal to about 25% of Haiti’s GDP, and could theoretically provide each person affected by the earthquake with about 9 months’ worth of a Haiti-average income, or could provide each person left homeless with 1.6-2.4 years’ worth of a Haiti-average income.
  • The total value of the damage could easily be greater than the total amount raised/pledged.

Considering the extent of the damage and logistical challenges, we wouldn’t necessarily expect all of the damage to be repaired, even 5 years from now. Therefore, we shouldn’t be surprised to see continued suffering and challenges in Haiti, and we also shouldn’t be surprised or upset that organizations are holding much of the money they’ve raised, for later rebuilding efforts. However, given the large amount available (and the large amount spent) in the context of Haiti’s economy, we should expect to see major and tangible benefits coming from the relief effort.

Details of the numbers above follow.

How much has been given?

The most comprehensive-seeming source we’ve found for disaster relief funding is the OCHA list of all humanitarian pledges, commitments & contributions in 2010 (XLS) found in the collection of 2010 Haiti-related documents on Reliefweb (“OCHA” for the remainder of this post). OCHA gives a total of about $3.5 billion committed plus another $1 billion in uncommitted pledges.

However, this isn’t the whole story – it seems to us that the OCHA data is missing a substantial amount. We compared the OCHA numbers to Chronicle of Philanthropy figures from a July 2010 survey. When there was a charity that we were confident we could identify in both tallies, we put the Chronicle figure for that charity side-by-side with the OCHA figure (see the “OCHA vs Chronicle – diffs” sheet in our analysis (XLS)). Even though the OCHA figures are as of January 2011 while the Chronicle figures are as of July 2010, the Chronicle figure exceeds the OCHA figure for the vast majority of matched charities, sometimes by a very large amount. Just looking at the charities that we can confidently match up between the two implies over $700 million in uncounted funding in the OCHA tally.

These matched charities tend to be the big ones, and collectively account for ~$1.37 billion out of the ~$1.65 billion total for all the figures in the Chronicle tally. (Note that we are using “worldwide funds raised” figures when available; using only U.S. figures would generate a total of $1.3 billion, consistent with the Chronicle’s own figure for the total). Furthermore, the funding that’s come in between July and now doesn’t seem significant – the latest Chronicle total for U.S. giving is $1.4 billion, compared to $1.3 billion in the July report.

So we’d say that the total amount of money given or pledged appears to be at least $5.2 billion (~$3.5 billion given according to the OCHA report; ~$1 billion pledged from OCHA report; ~$700 million in uncounted funding that we identified in the Chronicle tally). More thoroughly counting all the “missed funding” for particular nonprofits could raise this total, though probably not by very much.

The recent Chronicle update also states that about 38% of the funds raised have been spent. This applies to the charities in the Chronicle tally, not to all the agents included in the OCHA tally, but we can estimate the amount spent by multiplying our estimate of the total amount committed (not pledged) by 38%, to get $1.6 billion spent.

Have charities raised as much as they expected/hoped to?

Here’s a comparison of how much a few large charities said they needed in February 2010 (from a Chronicle survey) vs. how much they had raised as of July 2010 (from another Chronicle survey).

Charity Requested as of 2/21/10 Received as of 7/9/10
Partners in Health At least $100 million $85 million
Oxfam At least $100 million $90 million
Save the Children $85 million to $115 million $71.4 million
UNICEF $128 million $73.1 million (US only)
World Vision At least $100 million $192 million

Note that the “Requested” figures above are multi-year appeals.

We don’t want to read too much into this data. The announced “funding needs” may have been formulated more based on what was achievable (i.e., trying to set a “stretch target”) than on what expenses were expected – we don’t know. But the basic picture seems to be that major charities raised close to what they were seeking in the few months after the disaster.

How much has been spent/raised in the context of Haiti’s economy and the damage it suffered?

According to the CIA World Factbook, Haiti had an estimated 2009 GDP of $6.56 billion and a population of about 9.6 million, which implies an average annual income (in U.S. dollars, not adjusted for purchasing power parity) of $680.

$5.2 billion (the amount raised/pledged) is equal to about 80% of Haiti’s total yearly economic output; $1.6 billion (the amount spent) is equal to about 25%.

Using an estimate that 30% of the population was affected by the earthquake (see below), $5.2 billion (the amount raised/pledged) would theoretically be enough to give each affected person about $1700, or 2.7 years’ worth of Haiti-average income (though people affected by the earthquake were largely in urban areas and thus probably had above-average incomes for Haiti). $1.6 billion (the amount spent) would be enough for $533, or about 9 months’ worth of Haiti-average income.

Using an estimate that 1-1.5 million people were left homeless by the earthquake (see below), $5.2 billion would theoretically be enough to give each person left homeless about $3500-$5200, or about 5.1-7.7 years’ worth of Haiti-average income. $1.6 billion (the amount spent) would theoretically be enough to give each person left homeless $1067-$1600, or 1.6-2.4 years’ worth of Haiti-average income.

That wouldn’t necessarily be enough to compensate for all the damage, especially in a logistically challenging situation. One study (PDF) estimated that the earthquake caused $7.2 billion-$13.9 billion worth of damages, which is 1.5-3x the amount of aid given and 1.1-2x the value of Haiti’s annual GDP. We find this study highly unreliable (it is attempting to predict the value of the damage based on a linear regression using a very small number of predictive variables such as the number of people killed, land area, population, GDP and disaster type – see page 6), but we do find it plausible that the total losses for a person left homeless could exceed 8 years’ worth of their income, and thus we find it plausible that the amount of aid raised/pledged is not equal to the value of the damage caused.

A note on figures for “people affected” and “people left homeless”

Unfortunately we haven’t yet been able to find official government or U.N. estimates for these figures, accompanied by details of the estimation procedure; the best we’ve found is various news stories citing the government or nonprofit agencies. The number we’ve seen for “total people affected” is 3 million (citing the International Federation of the Red Cross) and the numbers we’ve seen for “people left homeless” are 1 million (citing the government) to 1.5 million (we’ve seen this number in several places including the Chronicle of Philanthropy but haven’t seen a source for it). This would imply that the earthquake affected about 30% of Haiti’s total population and left 10-15% of the total population homeless (using the CIA World Factbook population estimate of 9.6 million).

Assessing disaster relief organizations

(updated on: ) | by Holden

Over the past year, we have been examining disaster relief organizations, with particular attention to the quality of the information they provide on their work in the 2004 Asian tsunami and the 2010 Haiti earthquake.

We don’t think disaster relief is the best cause for impact-oriented donors and we are unable (details below) to apply our standard international aid criteria to disaster relief organizations. However, we think there is a lot of room for improvement in the information currently available for disaster relief donors, and a lot of room to improve incentives for the major disaster relief organizations to be as transparent and accountable as possible.

Questions we’ve focused on

In examining charities’ information, we’ve focused on the following questions:

  1. Financials. Did organizations clearly and frequently disclose how much they were seeking, what they’d do if they raised more than they sought, how much they’d raised and how much they’d spent?
  2. Transparency on activities. Did charities provide specific and comprehensive accounts of their activities in the affected areas? Could someone on the ground verify or refute their claims about how its money was spent?
  3. Results. Did charities publish substantive information on their successes and shortcomings?
  4. Everyday work. Do charities publish clear and substantive information about their non-disaster relief work? We’ve argued before that donations intended for disaster relief may effectively fund charities’ everyday operations instead (even if they are formally earmarked for and allocated to disaster relief). Therefore, we think the quality of (and transparency around) a charity’s everyday work ought to carry a heavy weight in donors’ decisions.

What we’ve found

  1. Financials. Most major organizations have disclosed how much they’ve raised and (less frequently) how much they’ve spent. No organizations have been consistent and clear about how much money they were seeking. In cases where we have been able to find posted amounts sought, the charities posting them generally quickly raised more than they were seeking. In one case the appeal was constantly revised upwards. Despite raising much more than they sought (and than they’ve spent), we’ve only seen two groups that stopped taking donations for the disaster.
  2. Transparency on activities. Few organizations are clear about what they’re doing on the ground. Most give examples and very broad budgets. But with a few exceptions, charities have not given specific enough accounts to givea sense of how they spent their money.
  3. Results. No evaluations seem to be posted yet for Haiti relief. Many have been posted for work on the Asian tsunami; we haven’t yet had time to evaluate these evaluations (and don’t expect to have done so by next week).
  4. Everyday work. Few of the large organizations we examined are clear about how their overall budget breaks down and what they do around the world. There are some groups that are clearer than others.

How we chose organizations to examine

We are unable to predict the location and nature of the next disaster, so it wouldn’t make sense for us to look for an organization with specific ongoing, outstanding, underfunded work (as we usually do). Instead, we wish to focus on the large, global organizations that will probably attract the lion’s share of donor attention and funding whenever, wherever and however the next disaster strikes. To create a list of these organizations, we referred to:

  • The Chronicle of Philanthropy’s tally of which charities had raised how much for Haiti relief as of July 2010. The biggest numbers on this list are likely the biggest money-raisers overall; charities absent from the list likely didn’t raise enough to be of interest, or didn’t share their tallies with the Chronicle (and we feel the latter would be a significantly bad sign for accountability).
  • A list we had made in the weeks immediately following the Haiti earthquake, noting which charities were advertising via Google Adwords for earthquake-related searches. This is a simple heuristic for finding charities that are likely to rely upon, and solicit, donations from the public at large.

Specifically, we rated and ranked any charity that either (a) appeared in the Chronicle tally and advertised via Google Adwords; or (b) was among the 10 biggest money-raisers in the Chronicle tally, regardless of whether we saw its ads on Google Adwords. These criteria produced a list of 19 prominent charities; we added a few more at our discretion.

More information forthcoming

Next week we will be publishing:

  • Detailed writeups on each charity we examined and the information we could find on the above questions.
  • A summary table showing the strength of different charities’ answers to our questions, and noting the charities we think stand out for their superior transparency so far.
  • Full details of our process for creating these pages.