I started work at GiveWell six months ago, just a few weeks after graduating from college. I had been following GiveWell pretty intensely for more than a year, since I had gotten back from my own trip to India. During that time, I had become a little obsessed: I had read the entire history of the blog and got really excited each time GiveWell finally posted the audio from the most recent board meeting.

Even as a serious GiveWell fan, though, there were a number of things that I didn’t know about the organization that I should have. These aren’t secrets or titillating stories about office politics, just some things that I’ve learned that I didn’t know before.

The biggest challenge remains “find outstanding giving opportunities” - not “get more eyeballs.” I wasn’t totally ignorant about the difficulty of finding outstanding giving opportunities, but I thought that GiveWell was clearly doing so better than anyone else working publicly, and that accordingly it should focus more on outreach, rather than improving research. As an outsider, I didn’t have a good sense of how much went into the recommendations or all the work that goes into charities that don’t end up receiving recommendations. I didn’t think it was easy, but it seemed like Holden and Elie pretty much had it under control, and that there was lots of low-hanging fruit on the outreach side.

As far as I can tell now, neither of those things are really true.

On the outreach front, GiveWell had already tried or looked into many different strategies, even if they hadn’t blogged about it. And because our users generally aren’t typical donors, a lot of the things that charities normally do to cultivate donors might be actively harmful for us. (But we definitely haven’t thought of everything, so please do let us know or comment if you have ideas for how we could “sell” our research better.)

On the research front, although it isn’t very hard to come up with better recommendations than other charity evaluators, we face two problems I hadn’t fully considered:

• Room for more funding. Because a number of large funders are scooping up excellent funding opportunities in global health, many good chances to help people are already taken. We need to find charities that are good bets, but not so obviously good that they have all the funding they can productively use.
• Our competition isn’t other charity rating organizations. This is about the baseline that GiveWell’s recommendations are compared to, rather than the competition for funding opportunities. For a long time, it has seemed natural for people to compare GiveWell to Charity Navigator or Philanthropedia, but as we continue to grow, I think that comparison becomes less and less salient.

  As we raise our ambitions with projects like GiveWell Labs, we will be “competing” not with other charity evaluators but with foundations. Because some foundations are extremely strategic, well-resourced, and focused on the same goal of doing as much good as possible, finding better giving opportunities than they do is a much higher burden.

Both of these problems become harder as GiveWell grows, because we’ll need to create more “room for money moved” and will more naturally be compared to foundations rather than other charity evaluators.

Read the rest of this entry »

We ask you, as a donor, to turn down some great pitches – “Your interest-free loan will help this person escape poverty forever,” “You can give a cow to a poor family for Christmas,” etc. – and give instead to charities that aren’t terribly good at storytelling. Why?

It comes down to this. We think that most of those stories are just that – stories. (For more, see our summary of recent posts on “big-name” charities, which we feel are representative of the full set of charities we’ve reviewed.) But if you give to one of our top charities, you really can save (or dramatically change) a human life.

It hasn’t been easy to find charities that we can honestly say this about. That’s what our process is built around and where most of our energy goes. This week we’ve blogged about the best we’ve found, VillageReach and Stop Tuberculosis Partnership. There is plenty of room for doubt even with them, but overall we think there is a strong case - even for the skeptic - that your donation to them can save a life.

What do we mean when we say “save a life?”

By “you can save a life,” we don’t mean anything as simple, concrete, or easy to grasp as the stories charities usually tell.

• Your gift can’t literally be linked to an individual. It will help an organization that, all things considered, is achieving a lot of impact for what it spends.
• If you must know what “your” dollars are doing, it’s likely that they’ll be sitting in reserves to ensure financial stability, or enabling a slightly larger travel budget for evaluators, or something similarly unexciting.
• It’s even highly possible that your donations will be wasted, and that the charity you give to – even the best you can find – will fail. We don’t think there are true guarantees in aid.
• Even if these charities are succeeding, it’s very likely that your donation won’t ultimately result in the charity doing anything differently. It’s pretty hard to think about how $1000, by itself, could really change anything about Stop TB Partnership’s plans for next year.
• Yet donations add up. 50-100 of these donations could mean a significantly larger grant, more people getting tuberculosis treatment … and that could mean families staying intact instead of being struck by sudden death.

The truth is that it takes a lot of abstraction and analytical thinking to really think about how your donation saves a life. The life you can save is an “expected” life (”expected” in the sense of probabilistic expected value) - it isn’t a person we can point to or show you a picture of. More than with typical charities, you have to use your imagination. But more than with typical charities, your impact is real.

With opportunity comes responsibility

In The Life You Can Save (which prominently features GiveWell and which we have reviewed), Peter Singer writes:

By donating a relatively small amount of money, you could save a child’s life … we all spend money on things we don’t really need, whether on drinks, meals out, clothing, movies, concerts, vacations, new cars, or house renovation. Is it possible that by choosing to spend your money on such things rather than contributing to an aid agency, you are leaving a child to die, a child you could have saved? (pg 5)

Our corollary: is it possible that you are leaving a child to die when you choose to donate to a charity with a “feel-good” story rather than a charity with a great case for real impact?

It is true that, as our critics often point out, a charity can be impactful without being demonstrably impactful. But when one charity proves itself and another leaves you guessing, it seems clear to us which one offers the “better bet” – and more “expected lives saved” – given the information available. When you have the option of giving to an outstanding charity that demonstrably can save a life, how do you justify giving to a charity whose true impact is essentially a big question mark?

I’ll leave this blog’s last words for 2009 to Natalie, a relatively recent GiveWell hire (she has been working full-time on research since July).

Sometimes I’m almost tempted to give to a charity I know less about. I’ve been over VillageReach and I’ve seen how complex the situation is and how many questions there are. If I gave to some charity I know nothing about, I could just think about the story they tell and feel good and not have these nagging doubts. But I’m not going to do that – in the end it’s more important to me that I really make a difference.

Normally, we focus on identifying outstanding charities, and minimize the time spent on opaque or otherwise lackluster ones. But lately, we’ve gone into a bit more detail about our take on several of the best-known and most appealing charities out there.

What all of the charities below have in common is that (a) we have major questions and concerns about their activities; (b) the information necessary to see how serious these concerns are does not seem to be available. (In most cases our assessment is based on significant back-and-forth with the charities themselves, though in some cases we are going off their website.)

• Kiva: commonly perceived as a way to (a) make interest-free loans to (b) entrepreneurs you personally select. In fact, it is neither. (See GiveWell Board member Tim Ogden’s summary of the recent discussion/controversy.)

  We ultimately see a Kiva “loan” as a gift to a microfinance institution, and we have major concerns about supporting any but the most outstanding microfinance institutions as well as concerns about Kiva’s due diligence process specifically.

  Full set of blog posts on Kiva

• Grameen Foundation and other large U.S. microfinance charities: in addition to our general concerns about microfinance charities (and the fact that these charities have not provided the information that would address them), we find that these organizations are rarely even clear about what their value-added is. Also see the disappointing (and eye-opening) evaluation of the Village Phone program.

  Full set of blog posts on Grameen Foundation

• Heifer International: commonly perceived as a way to “give a cow to a poor family as a gift,” but this is in fact a donor illusion -donations support Heifer International’s general “agricultural assistance” activities. We have concerns about this general area and concerns about giving livestock specifically. Neither Heifer’s website nor its grant application have provided the kind of information needed to address these concerns.

  Full set of blog posts on Heifer International

• Smile Train: Prof. Steven Levitt of Freakonomics sees it as using an innovative model to deliver cleft surgeries for $250 apiece. Our analysis suggests that it has “more money than doctors” and that it is therefore out of room for more funding. Smile Train’s recent removal of the charts we questioned and its own explanation of why it is giving grants to other major charities appear to support our analysis. In addition, we have major concerns about surgery charities that Smile Train has not provided enough information to address.

  Full set of blog posts on Smile Train

• UNICEF offers donors the chance to “buy” vaccines and other items. We suspect that this offer is a donor illusion, but the larger issue is that UNICEF provides almost no information at all about its huge variety of activities and how they are going. Most giant international aid charities (Save the Children, World Vision, etc.) are similarly complex and opaque, with CARE coming the closest to meaningful transparency.

  Full set of blog posts on UNICEF

• The Acumen Fund stresses its metrics-driven approach to “patient capital,” investing in businesses that aim for long-term sustainability (i.e., profitability) and social impact. As of now, we see no substantive available information about its track record.

  Full discussion

• The Robin Hood Foundation also stresses its emphasis on metrics, quantification, and analytical thinking. But we have found that it is unwilling to share any information about the results of its grantmaking. (Note that it also pushes the “0% overhead” donor illusion.)

  Full set of blog posts on The Robin Hood Foundation

• The Millennium Villages project states that it is “demonstrating that it’s possible to escape … extreme poverty” (front page of its website). As far as we can tell, very few results have been demonstrated to date, with a few boosts in crop yields as its main evidence of impact - evidence we feel is highly questionable.

  Full discussion

• The Worldwide Fistula Fund has been praised by Nicholas Kristof as “giving new life to the pariahs.” Our assessment is that the Fund is pushing a debatable agenda whose effectiveness has yet to be shown.

  Full discussion

• The Carter Center is, in our view, by far the “best bet” (for an individual donor, our focus) of the charities listed in this post. It has achieved substantial, demonstrable impact in the past through its work on guinea worm eradication, and unlike all of the charities listed above, it publicly discloses substantive technical information on the impact of its past activities.

  However, there is a crucial piece of information missing: how it allocates its budget between programs (in particular, how much it spends on its stronger and better-documented vs. more questionable programs). It has declined to share this (basic, in our view) information in repeated exchanges.

  Full set of blog posts on The Carter Center

We think the above charities are fairly representative of “average” charities in international aid. Some tell better stories than others and some have more disclosure than others. But in almost all cases, international aid charities are (a) carrying out complex projects that can fail to do good (or even do harm) in a variety of ways, and (b) not systematically sharing the information that would make it possible to assess how their work is going.

GiveWell is devoted to finding charities in which we can have more confidence. We’ll be discussing our two top-rated charities working internationally in forthcoming posts.

If a charity demonstrates that its core program has changed lives in the past, is likely to change lives in the future, and gets great “bang for your buck,” is this enough reason to donate to it? We say no.

The missing piece: Will more funding lead to more of the good program(s)? We generally call this the “room for more funding” question, and we’ve seen next to no helpful discussion of the issue within academia, within the nonprofit sector, or anywhere else.

Often, when I raise this issue, the response I get is “But is that a real problem? Are there charities that have great programs they can’t or won’t expand with more funding?” The answer is yes. Examples:

• Our analysis of Smile Train strongly suggests that its core program of directly funding doctors has “more money than doctors.” Thus, over 50% of Smile Train’s funds go to activities far from what their fundraising focuses on, including grants to other organizations, research, and “provid[ing] materials on cleft lip and palate for free to anyone interested in this birth defect.” Perhaps these activities have value, but it would be a mistake to donate to Smile Train just based on their headline program.
• The Aravind Eye Care System is one of the more impressive humanitarian organizations we have seen, performing vision-restoring surgery extremely cost-effectively. They have been so successful, in fact, that their core program doesn’t need donations - as they have explicitly told us. Revenue from for-pay surgeries subsidizes free surgeries, and donations are used on entirely and substantially different programs such as distribution of spectacles and free food.
• Today’s Aid Watch post gives an excellent picture of why it’s so important to be wary:
  

  according to Fred Martin, Communications Director at CHF, “In fact our Food Pak program is a small portion of what we do. We highlight it because it is our flagship program that we’ve seen work very well in building relationships with the poor so that deeper needs can be uncovered and responded to.” I learned from Fred they also provide beds in eastern Europe and medicines in Asia …

  As long as charities can get away with it, their incentive is to advertise the best program they have, even well beyond the point where that’s the program that needs more money.

There can be many bottlenecks to expanding a program besides money (skilled labor, environments that are conducive to the program, etc.) If you want to fund great programs, you have to ask not just “What have you done and has it worked?” but “What will you do with more funds than you’re currently expecting?”

We haven’t identified any easy answer or simple formula for this question. We believe that “restricting” your donation to the program you favor is generally a futile endeavor (more on this in a future post).

We have developed some relevant ideas. In addition to some rules of thumb for avoiding the most tangled cases, we ask the strongest charities for documents that speak to the “room for more funding” issue directly, such as examples of un-funded but strong project proposals and financial “scenario analysis” (details to come in future posts).

However, we have found that requesting such documents is an uphill battle because the request is generally so foreign. You won’t see financial scenario analysis on any standard list of “documents a charity should be sharing” (from the IRS or anyone else).

Foundations arguably don’t need to deal with the challenge discussed here, because they can give money in large enough chunks to dictate which projects get carried out. (The extent to which this practice is wise is another question). This may be why no one else seems to be asking for information on “room for more funding.” Whatever the reason, it’s an issue that needs much more attention than it’s getting.

GiveWell spends a lot of time on the question, “Should I give to charity A or charity B?”

One of the things that has surprised us about the world of charity is how many people insist on answering, “Both” or “You can’t/shouldn’t be asking that question.” To them, all that matters is whether a charity does some good, not how much good it does or how it compares to other options.

One statement of the idea comes from none other than the Jeffrey-Sachs led WHO Commission on Macroeconomics and Health (footnote 24):

Many have asked the Commission what to do if the donor money is not made available—in essence, how to triage with less money. We are asked to prioritize millions of readily preventable deaths per year, since we have already narrowed our focus to a small number of conditions that have an enormous social burden and that have low-cost interventions that are at least partially effective. Not only is this kind of triaging ethically and politically beyond our capacity, but it is also exceedingly hard to do in sensible way. Those who hope for a simple answer, for example to focus on the cheap interventions (immunizations) while putting off the expensive interventions (higher-cost prevention programs and antiretroviral therapy needed to fight AIDS) to a later date, misjudge the practical choices we face. The AIDS pandemic will destroy African economic development unless controlled; to fight measles but not AIDS will not begin to meet Africa’s human and economic needs. It would be wrong to go to the other extreme as well, and let the legitimate need to fight AIDS end up starving the cheaper interventions, so we advocate both. Moreover, the infrastructure developed to fight AIDS will support the infrastructure needed to fight measles, especially if strengthening such complementarities is explicitly built into the AIDS control effort. It is vastly more fruitful to design and finance a comprehensive program that addresses many critical health needs than to pick and choose the apparently inexpensive items.

More examples of this mentality can be seen in recent comments to our blog by John, Yael and Steve.

Our response to these comments is simple. Most donors have a certain amount they are willing to give, and that amount is less than enough to fully fund even one of the world’s underfunded humanitarian initiatives.

We don’t believe that any problem that people suffer from should be ignored or trivialized. We do believe that individual donors must choose which to address, and that they should be factoring in questions like “What evidence is there that my money can even do anything to address this problem?” and “How cost-effectively can I create change by giving my money here?”

In their personal and business lives, people constantly make tradeoffs on a limited budget. Imagine if a salesman tried to argue, “How can you choose between this car and that one? They’re both wonderful cars and each would make the other more useful. You should buy them both!”

Yet in the world of charity, the basic and undeniable idea of triage is constantly denied.

The New York Times recently profiled Chess in the Schools:

The Chess-in-the-Schools program has sought to foster analytical skills on the theory that these will help students succeed academically. The group teaches 20,000 children a year and calculates that it has taught 425,000 children since 1986. Children gather to learn the game at the group’s headquarters in Manhattan.

It seems like 20 years and 425,000 children is quite a lot of investment in the “theory that [chess] will help students succeed academically.” The Times feature provides a calming justification for the investment: “Chess helps promote intellectual growth and has been shown to improve academic performance.” Let’s look at the evidence for this claim.

The study we found

An early-1990s study looks at achievement test scores of chess-playing students over two years at District 9 in the Bronx. It observes that (a) the overall average reading score improved among chessplayers by about 5 percentile points, but didn’t improve among the set of remaining District 9 students; (b) 15 of 22 second-year participants improved their reading scores by some amount, while only 491 of 1118 non-participants in the district - and 245 of 655 non-participants with high reading scores, improved.

This study is riddled with major problems:

• The numbers the researchers choose to compare seem arbitrary and possibly cherry-picked. Why do the researchers look at the “percentage who improved” among second-year chessplayers but not for both years? Why do they compare the second-year students to “high-performing nonparticipants,” but not give the same comparison when looking at all students?
• The problem of selection bias is unusually obvious here. They’re comparing kids who volunteered to play chess against those who didn’t. Think of the chess club members at your school, and ask yourself if they would have been just like all the other kids had chess club not been offered. There’s no reason to think these two groups of kids are otherwise similar or would be expected to respond similarly to school.
• This is a study of somewhere between 22 and 53 students at a single district in the early 1990s. Even if the study were highly rigorous, it would still be a long way from “proof that chess helps promote intellectual growth.”

The studies we couldn’t find

The Chess-in-the-Schools website states:

In 1991 and 1996, Stuart M. Margulies, Ph.D., a noted educational psychologist, conducted two studies examining the effects of chess on children’s reading scores. The studies demonstrated that students who participated in the chess program showed improved scores on standardized tests. The gains were even greater among children with low or average initial scores. Children who were in the non-chess playing control group showed no gains.

Another study in 1999, measured the impact of chess on the emotional intelligence of fifth graders. The results of the study were striking. The overall success rate in handling real life situations with emotional intelligence was 91.4% for the children who participated in the Chess-in-the-Schools program. In contrast, those who were not involved with the chess program had an average overall success rate of only 64.4%.

We’re guessing that the study we’re looking at is an update of the 1991 study since it references no previous studies and discusses results from 1991 and 1992. We can’t find the other studies anywhere. Chess-in-the-Schools provides neither links nor citations.

Even in the best-case scenario, it’s apparently been at least a decade since the last test of the Chess-in-the-Schools model.

“Chess helps promote intellectual growth and has been shown to improve academic performance?”

In researching charities, one of the more discouraging things we’ve learned is how little support it takes for a statement like “Chess helps promote intellectual growth and has been shown to improve academic performance” to be repeated by charities, donors, and even the media.

As far as we can tell, Chess-in-the-Schools is not a demonstrated success story. It’s just been promoted and scaled up like one.