The GiveWell Blog

Self-evaluation: GiveWell as a donor resource

(updated on: ) | by Holden

This is the first post (of four) we’re planning to make over the next two weeks focused on our self-evaluation and future plans.

This post answers the first section of questions we posed to ourselves in January about the state of GiveWell as a donor resource. For each question, we discuss

  • Our progress over the last year (specifically, since our last business plan in 11/2008);
  • Where we stand (compared to where we eventually hope to be);
  • What we can do to improve from here.

Does GiveWell provide quality research that highlights truly outstanding charities in the areas it has covered?

This is in some ways the most difficult question for us to give a meaningful answer on. “Quality” and “outstanding” are concepts that depend a lot on your worldview, so when we review the research that we ourselves have created, you can expect our opinion to be skewed toward the positive. That said, we can discuss how our research compares to what we feel it would ideally be.

At this point we are satisfied with the quality of our international aid report, but see a need to cover more causes and to be more systematic about getting meaningful critical feedback.

Progress since 11/08

As of 11/08,

  • We had published research on U.S. equality of opportunity and international aid, but felt our research process had substantial room for improvement (among other things, we felt it relied too much on the arbitrariness of how charities responded to our grant application, and that it didn’t make enough use of publicly available independent analysis). We planned to “re-do” the cause of international aid, with substantial changes to the process (see our June 2008 plan).
  • We had essentially nothing in the way of critical feedback on our research, aside from very high-level conversations about our conclusions at Board meetings.

Since then,

  • We have published our 2008-2009 international aid report using our revised methodology. We feel that we have found an appropriate way of investigating charities without relying on grant applications and that we have substantially improved on our prior recommendations (see our top charities – the two three-star charities were found this year, while the top charities from last year’s report have two-star ratings). We feel that our revised methodology is the best we can do with the resources and information available, and do not feel a need to “re-do” our coverage of international aid (although we will need to keep it up to date).
  • We also completed a grant application process for economic empowerment, whose results we will be publishing shortly.
  • A small number of people have scrutinized our research in detail and had in-depth conversations with us, partly in private and partly in public. These include (a) Board member Jon Behar, with whom we publicly discussed our reasoning for our two top-rated charities; (b) Phil Steinmeyer and Ian Turner, both donors whom we had no pre-GiveWell connection to and who put substantial time of their own into this year’s giving decisions (In each case, a substantial amount was ultimately allocated to our top-rated charity, VillageReach); (c) 5-10 other individual donors who have gone over much, though not all, of our research and discussed it with us at length.
  • We have also engaged in public conversation at a more general level via our blog. There have been generally favorable reactions – though these are not the same as endorsements – from people with substantial experience and/or publications on foreign aid (including David Roodman, Saundra Schimmelpfennig, Alanna Shaikh and Chris Blattman).

Where we stand

Internally we are satisfied with the quality of our research, but:

  • We need to add in direct observation of charities’ observations, to the extent that we can.
    • For international aid, it isn’t practical to get both deep and representative field experience with each, or really any, of the charities we review. Gaining truly representative up-close experience with the operations of even a single small charity (let alone one like PSI) could take years; gaining representative up-close experience with several charities, for the sake of comparative analysis, is in our view entirely impractical (and so it is not surprising that no such comparative analysis appears to exist).

    • That said, at this point we have not visited a single developing-world charity, and could potentially gain a lot from a site visit.
  • Our research needs more substantial external checks than it has gotten to date. The only people who have scrutinized it in detail (as opposed to discussing/linking to it at a general level) are, like ourselves, outsiders in the field of international aid. Though people aware of our research (including ourselves) generally seem to consider it high-quality, it is important that we eventually subject it to strong, critical scrutiny from people with substantial relevant experience and credentials.

What we can do to improve

  • We can conduct site visits to one or more of our reviewed charities. I am currently planning a trip to Africa from 2/10-2/23 during which I will visit both VillageReach and the Small Enterprise Foundation.
  • We can make a concerted effort to subject our research to strong, critical scrutiny from people with substantial relevant experience and credentials. The first basic step – which we haven’t yet taken – is simply to systematically identify the people who are most likely to be both qualified and willing to review our research, then make persistent contact with them (both personally and through contacts of ours) asking them specifically for in-depth reviews of our research. If they decline, we’ll hopefully learn more about why they are declining and what we can do to make reviewing our research more worth their while.

Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?

We believe that our work has become substantially more usable and practical to engage with. The limited external checks on our research (discussed above) are a major potential obstacle for donors to evaluate it.

Progress since 11/08

A year ago, the poor design of our website was a major potential obstacle for donors seeking to understand our work. Phil Steinmeyer summarized the problems in an email earlier this year. A donor who had the time and inclination could follow every step of our process, but otherwise had very little to go on besides a list of top charities and stories of our project from us and the media. Since then, we have:

  • Substantially revamped our website (not just its design but its organization).
  • Made a concerted effort to “repackage” much of our research in an extended series of blog posts. While our website focuses on presenting our recommended charities and the process we followed to identify them, these blog posts present (a subset of) our research such that particular general-interest points can be followed without having to read through a lot of other context. Comments on the blog posts also can give context on how others view our research.

We have also made some progress in terms of giving donors ways to evaluate our credibility:

Where we stand

A donor with the time and inclination can follow our full process, reasoning, and sources, and (as discussed above) a few that we know of have actually done so. A more casual donor has some, though limited, ways of assessing our research:

We feel these options are insufficient on the whole, given our goal of influencing as many donors as possible. We feel that we should have a single, easy-to-find roundup of available information on the reliability of credibility of our research.

What we can do to improve

  • We can make a concerted effort to subject our research to strong, critical scrutiny from people with substantial relevant experience and credentials, and to create public records of such scrutiny. (This is also a key area for improvement in the previous section, “quality of our research.” We feel that strong critical scrutiny would improve both the actual quality of our research and donors’ abilities to gauge its credibility.)
  • We can put together the best possible consolidated case for the credibility of our research.

Has GiveWell covered enough areas to be useful?

Our original business plan states:

We believe that pitting charities against each other using concrete, consistent criteria is the best way to evaluate them in a way that is thorough and understandable to others. At the same time, we do not want to compare all charities in the same terms, because this would involve making philosophical decisions that should be made explicit rather than buried in conversion factors. We’ve therefore taken the approach of dividing charities into causes by broad philosophical goal … Charities are pitted directly against each other within causes, but not between them.

In other words, while we sometimes argue for one broad cause over another, we ultimately seek to serve donors with many different values. We are a long way from accomplishing this goal.

Progress since 11/08

The focus of the last year has been a report on international aid. The intent was to improve both our depth (i.e., quality of the research and top-rated charities) and our breadth (i.e., the number of “causes,” or different value sets, we can make recommendations for). Our breadth did not improve as much as we had hoped, because we concluded that health has by far the strongest options for donors (and the charities we had previously recommended were in the area of health). We have been able to make only weak recommendations for education and economic empowerment (although better recommendations for the latter are forthcoming as a result of our recently completed grant application process ).

Where we stand

Currently, the only charities we recommend with two-star ratings or higher are in the areas of U.S. education (K-12), U.S. early childhood care, and global health. This is enough to serve some donors, but overall gives quite a narrow set of options.

What we can do to improve

We are considering research on areas including:

  • More sub-causes within international aid – including disaster relief/recovery, charities aiming to help orphans and vulnerable children, and more attempts to find a strong microfinance option.
  • U.S. equality of opportunity – our current research on this cause comes from our original (2007-2008) research process. It should be re-done with the revised research process we used to create our current international aid report.
  • Disease research funding (e.g., American Cancer Society).
  • Environmental issues, particularly global warming mitigation.

There are many more charitable causes, but these are the ones we feel it would be most productive to research in the relatively short term (more in a future post).

Bottom line on GiveWell as a donor resource

  • Our international aid report meets our internal quality standards, but should be subjected to more systematic, in-depth, public scrutiny to improve both its quality and the ability of casual donors to judge its credibility.
  • We feel that both our international aid report and our report on U.S. equality of opportunity (which does not meet our internal quality standards) are valuable resources for donors interested in those causes, and more directly useful for impact-focused donors than any other available resources.
  • We offer a very narrow set of causes for donors and should research many more areas.

Economic empowerment grants awarded

(updated on: ) | by Holden

We have completed the process of allocating $250,000 in grants for economic empowerment organizations in sub-Saharan Africa.

49 organizations applied for our grant, out of 157 invited. From these, we felt that two stood out in making a strong, evidence-based case that they are improving the financial situations of low-income people in the developing world. We have awarded $125,000 to each.

The Village Enterprise Fund is a microenterprise organization providing cash grants as well as business training and mentoring services to extremely poor business owners in rural Kenya, Tanzania and Uganda. As we have written before, we feel that cash transfers are a relatively direct and promising way of helping people, a reasonable default approach in an area where little evidence is available on what works. Village Enterprise Fund stands out for its clear targeting of extremely poor clients (and data collection to ensure that the targeting is working), addressing a key concern about cash grants.

The Small Enterprise Foundation is a microfinance institution in South Africa. We have written before that we see great promise in microfinance charity, but also have serious concerns about it; these concerns led us to investigate individual microfinance institutions overseas, rather than sticking with large U.S. charities. The Small Enterprise Foundation stands out for its ability to answer our key questions about microlending. (Note: SEF is not a U.S.-registered charity.)

Within the next few weeks, we’ll be publishing everything we’re allowed to publish about how this decision was made: which organizations applied, what materials they submitted, what criteria we used, and how we made the final call.

For now, see the audio and documents from the board meeting where we made the grant decision.

Haiti “room for more funding” at the organization level: Not enough information

(updated on: ) | by Holden

In a previous post, we asked whether Haiti earthquake relief has “room for more funding” and concluded that, in general, it isn’t clear. (For more on the general topic of “room for more funding,” see our 5-post series on the topic.)

Of course, it probably makes a big difference which organization we’re talking about. We’ve seen a lot of different charities soliciting funds in the context of the Haiti earthquake, and some of them may have greater abilities than others to translate funding into relief assistance. The problem for donors is that by and large, these charities aren’t accompanying their appeals with the information we’d need to have a sense of their “room for more funding.”

The key questions we feel charities should be answering

We assert that any organization raising funds in the context of Haiti earthquake relief should provide clear, prominent answers to the following questions:

  1. How much are you trying to raise?
  2. Roughly speaking, what activities are you seeking to fund?
  3. How much have you raised so far?
  4. If you raise more than your target, what will you do with the remaining funds?

Answers to such questions would make it possible to hold organizations accountable (well after the fact) for how they actually spent money raised in their relief appeals.

Charities’ answers to these questions

We’ve examined the online appeals of charities that (a) have raised $10 million or more in the context of Haiti relief, according to the Chronicle of Philanthropy’s tally or (b) have come up via Google Adwords on relevant searches (“Haiti”, “Haiti relief”, “Haiti donations”, etc.) (We also included Yele Haiti due to the attention it’s received, even though it did not meet either of these criteria.) It seems fair to say that all of these charities have done significant fundraising in the context of the Haiti relief effort.

For each, we looked on their website for answers to the above questions, both by clicking relevant links and by Google-searching the site. (Note that this work was mostly done last week.) We provide our results in Excel format.

Most (not all charities) provide at least an overview of what sort of aid they are looking to provide. However, only 6 out of 17 appear to give any target for how much they’re looking to raise; only three appear to be sharing how much they’ve raised to date (although many more appear to be sharing this information via the Chronicle of Philanthropy’s tally); and only two (the two mentioned in our previous post, Heifer International and Doctors Without Borders) are explicit about what they will do with “extra” funds.

We don’t believe that a top-down pooling of funds is the only way to make sure money gets to the organizations that can use it productively. We see potential in the idea of different charities’ raising funds for different efforts, as long as they make it clear how much they are looking for and why; make it possible to hold them accountable for how they spend the funds they raise; and are explicit about the point at which they’d spend funds on other potentially worthy causes. However, from what we can see, charities aren’t currently sharing enough information along these lines.

Does Haiti earthquake relief have room for more funding?

(updated on: ) | by Holden

Donors have given more than $560 million to charities “to help earthquake relief efforts in Haiti.”

How much of that money has funded/will fund earthquake relief efforts in Haiti? How much should?

Money isn’t the only thing needed to deliver relief

Reports from the relief effort have stressed logistical challenges, such as blocked roads and limited access for planes and boats. See, for example, this interview on Reuters AlertNet from the 18th:

” The capacity of the Port-au-Prince airport is about to be increased but it is still a small airport. It’s very congested … The seaport is not operational and needs to be fixed in the coming days … It’s a big traffic jam of vehicles carrying humanitarian assistance and carrying people who want to leave Port-au-Prince … A massive effort is needed for Haiti but it needs to be done in a coordinated way. We need more realism about how long it takes to get an operation of this magnitude in such an intensely complicated environment running. A feeding operation for 2 million people is the goal. We know that in every case it takes time in the beginning and more time when every structure on which we can rely has been so appallingly hit.”

We’re not sure of the extent to which these issues have improved since. But we think it’s important to note that they do not sound like the kind of issues that can be solved directly by more supplies or even more money.

In fact, it is easy to see how deliveries of supplies and arrivals of volunteers could make things worse, if they are not carried out in full consideration of conditions within the country. We speculate that charities that are desperate to help, but without a strong on-the-ground presence, could be dropping off supplies without a clear plan for their distribution, and thus worsening congestion. This dynamic is described (for the case of the 2004 tsunami) in a Global Post article (H/T Good Intentions Are Not Enough):

although officials didn’t request any medicine, they received 4,000 metric tons of it, or more than 4 pounds for each person in the tsunami-affected area. There were multiple-year supplies of antibiotics, and palette loads of drugs unknown to health care providers. Seventy percent of it was labeled in a language that locals did not understand … In the end, most of the drugs had to be incinerated — you can’t simply send such a stock to the dump, where it would seep into the ground water and create another health hazard. That cost donors and the Indonesian government millions.

Other charities may be reluctant to overspend on the relief effort, for exactly these reasons. Doctors without Borders, which has a major prior presence in Haiti (over $15 million spent there in 2008, according to its activity report), is explicit that it is no longer seeking to use additional donations for the relief effort:

We are incredibly grateful for the generous support from our donors for the emergency in Haiti … We are now asking our donors to give to our Emergency Relief Fund. These types of funds ensure that our medical teams can react to the Haiti emergency and humanitarian crises all over the world, particularly neglected crises that remain outside the media spotlight. Your gift via this website will be earmarked for our Emergency Relief Fund.

Just because charities are soliciting doesn’t mean they need more funding for Haiti earthquake relief

Heifer International, is explicit about the possibility that its Haiti appeal will be overfunded, leading to funds’ going to its other activities:

Funds raised in this appeal will be used in the recovery and rebuilding effort in Haiti in the wake of the earthquake. Any funds that exceed the level needed to provide relief in this rebuilding effort will go toward the disaster relief fund and for the entire mission of Heifer International.

Other charities may be less explicit. (We have argued before that even if donations are formally earmarked for a project, they may effectively end up funding the charities’ general activities.) Worse, some charities may in fact be soliciting for Haiti and intend to spend the money in Haiti, even if they can’t do so productively.

In an ideal world, charities that had no strong and underfunded presence in the relief effort would not be soliciting donations in the context of the relief effort. But it seems to us that this crisis (and the media coverage around it) arouses people’s emotions in a way everyday suffering can’t. That gives charities reasons to “capitalize on the opportunity” – reasons that may be

  • Straightforwardly selfish: fundraisers are evaluated by how much money they bring in.
  • Based on morality: donors want to give more to a relief effort that can be usefully spent, but they give less to other humanitarian initiatives than can be usefully spent.

This dynamic could explain why so many different charities are clamoring for donations under the heading of “Haiti earthquake relief” (details forthcoming in a future post).

Can you make sure your money is going to the Haiti relief effort? Should you?

There are strong arguments that the gift you make for Haiti, in the heat of the moment, would be better applied to (for example) a bednet distribution program in Africa.

However, in our view, the fact that there are many other worthwhile activities doesn’t absolve charities of responsibility for being clear about how funds are spent, and if anything it increases the responsibility of donors to understand what they’re paying for and how it compares to their other options for accomplishing good.

We believe that “room for more funding” questions do not get enough attention. In future posts, we’ll be discussing how those questions might be answered in the context of the Haiti relief effort.

Haiti earthquake relief seems less cost-effective than everyday international aid

(updated on: ) | by Elie

The disaster in Haiti – and the media coverage of it – pull at the emotions in a way that everyday suffering in the developing world does not. However, our rough calculations suggest that in fact, a donor can have a bigger impact for less money by funding top charities’ everyday activities to reduce unnecessary death and debilitation.

We estimate a “generous” cost-effectiveness figure for a donation to Haiti is by considering (a) the total amount given and (b) the total number of people affected by the disaster.

Total amount given: It’s hard to find definitive figures for the amount of money already donated to the relief efforts, but ReliefWeb provides what appears to be a reasonable lower-bound (i.e., conservative) estimate. As of January 27, ReliefWeb reports that $1.2 billion had been given or pledged to Haiti relief efforts. (See document A on ReliefWeb’s Haiti Earthquake: Appeals and Funding page.) Note that its numbers clearly do not include all charities or all revenue sources: for example, it lists only $2.8 million raised by Partners in Health, whereas the Chronicle of Philanthropy’s roundup – up as of two days ago but seemingly down at this writing – lists $40 million (and also includes many charities that don’t appear at all in Reliefweb’s stats.)

Number of people affected: Haitian authorities estimated that 1 million people were left homeless by the quake (according to ABC World News) and perhaps 3 million were “affected” (according to the Guardian). For the sake of our calculuation, we’ll assume a range of 1-3 million people affected.

Expenses per person affected: using the Reliefweb number (assuming that nothing is excluded from it and that no more money is forthcoming, both assumptions that clearly understate the funds) yields a current level of $395-$1,185 pledged or donated per person affected. (For some interesting context, we’ve seen estimates that the 2004 Indian Ocean Tsunami affected 2,000,000 people and donors gave approximately $14 billion — funding per person of $7,000.)

We estimate that our top-rated charities save a life for approximately that amount. While it’s clearly not possible to directly compare the impact of relief efforts to the impact of saving an individual’s life, our feeling is that saving an individual’s life is likely to have a significantly larger effect on that person than the future change likely to occur in the average life of a Haitian due to the relief efforts.

Can choosing the right charity double your impact?

(updated on: ) | by Elie

Reader Evan writes:

I’ve been thinking about how best to donate to Haiti, and I reviewed some of the materials on your website and found them pretty helpful and persuasive. So thank you! But then my law firm announced that it would match donations to the Red Cross or Doctors Without Borders. Given that, I think I have to donate to one of those orgs: even if my money would probably be better spent elsewhere, it’s hard to imagine that it would be more than twice as well spent. Do you disagree?

My intuition here is different than Evan’s. My guess would be that giving to one of our top-rated charities could easily accomplish more than twice as much good as supporting the efforts of the Red Cross or Doctors Without Borders in Haiti.

This guess is largely based on two factors:

  1. The large divergence in relative cost-effectiveness of different programs (which can approach a factor of 1,000, not just a factor of 2) combined with the reasonable position that disaster relief is not among the most cost-effective avenues for charitable funds.
  2. A back-of-the-envelope calculation for cost-effectiveness of efforts in Haiti which puts it well below the cost-effectiveness of our top charities.

In this post, I’ll look at the first factor. I’ll post more on the second issue in a future post.

Cost-effectiveness for different approaches to helping people varies widely

The most cost-effective programs are so much more impactful per dollar than other programs that a much smaller donation to a top program will likely help significantly more people. We’re careful about our use of cost-effectiveness figures, and the Disease Control Priorities Report’s (DCP) in particular (which we think constitute “best-case” scenarios rather than what a donor can expect from his donation), but we do think they give a reasonable basic sense of the differences between different kinds of programs.

Figures 2.2 and 2.3 on pages 41-2 of the DCP report provide cost-effectiveness estimates for many common programs charities run. (These are all presented using the $/DALY metric. For more information on what this is, see our overview for interpreting the DALY metric.) Some of the most cost-effective programs are deworming programs ($3/DALY), expanding immunization coverage ($7/DALY), and bednets to prevent malaria ($11/DALY).

Some of the least cost-effective (but common among charities) programs are improved water and sanitation to prevent diarrhea ($4,185/DALY), some types of maternal and neonatal care packages ($1,060/DALY), and Antiretroviral therapy to treat HIV/AIDS ($922/DALY).

These examples are not meant to demonstrate that the less cost-effective programs are necessarily less worthy, but they do illustrate that the impact per dollar a donor can expect from his gift can easily vary by 2-3 orders of magnitude, even under assumptions that programs are essentially being carried out as intended. Of course, if some programs are poorly executed or simply ineffective, the difference can be much larger still.

With that context, when choosing which charity to support, I wouldn’t trade much confidence-in-an-organization to merely double the size of my donation.

As we’ve discussed before, with limited information we’d tentatively guess that disaster relief funds are closer to the less-cost-effective end of the range rather than the most-cost-effective end. With that in mind, I’d guess that a gift to VillageReach or Stop TB could easily accomplish more than twice as muc good as a gift supporting the Red Cross or Doctors without Borders in Haiti.

The above is very general and though relevant, not at all specific to the situation in Haiti. In a future post, I’ll post more on some specifics regarding Haiti and why I think it offers further support to the notion that donors can accomplish more good by giving to our top charities, even if they give less.

Two other small notes

There are a couple other factors that contribute (though in a relatively small way) to my conclusion here:

  • It doesn’t seem appropriate to consider causing one’s company to give a donation to be equivalent to doubling one’s cost-effectiveness. The firm may have taken matching funds from a pool already allocated to charitable giving, or the partners may have given the funds to charities themselves. Even if the funds wouldn’t otherwise go to charity, the firm likely has another motive for giving, which should lead you to consider how this program differs from other embedded giving programs, which we think are of dubious additional value.
  • Giving to a charity because it has demonstrated effectiveness has the additional benefit of signaling to other charities that effectiveness matters to donors. A core belief of ours at GiveWell is that rewarding charities for effectiveness in changing lives will incentivize other charities to improve their programs to compete for those donor funds. Proactive giving (i.e., trying to choose the best charity available) furthers this dynamic; passive giving (choosing from a predefined list) hampers it. It’s also quite possible that in a very direct sense telling your company that you’ve chosen to give in this way could influence them to adjust their own giving towards more considered and effective charities.