Normally, we focus on identifying outstanding charities, and minimize the time spent on opaque or otherwise lackluster ones. But lately, we’ve gone into a bit more detail about our take on several of the best-known and most appealing charities out there.

What all of the charities below have in common is that (a) we have major questions and concerns about their activities; (b) the information necessary to see how serious these concerns are does not seem to be available. (In most cases our assessment is based on significant back-and-forth with the charities themselves, though in some cases we are going off their website.)

• Kiva: commonly perceived as a way to (a) make interest-free loans to (b) entrepreneurs you personally select. In fact, it is neither. (See GiveWell Board member Tim Ogden’s summary of the recent discussion/controversy.)

  We ultimately see a Kiva “loan” as a gift to a microfinance institution, and we have major concerns about supporting any but the most outstanding microfinance institutions as well as concerns about Kiva’s due diligence process specifically.

  Full set of blog posts on Kiva

• Grameen Foundation and other large U.S. microfinance charities: in addition to our general concerns about microfinance charities (and the fact that these charities have not provided the information that would address them), we find that these organizations are rarely even clear about what their value-added is. Also see the disappointing (and eye-opening) evaluation of the Village Phone program.

  Full set of blog posts on Grameen Foundation

• Heifer International: commonly perceived as a way to “give a cow to a poor family as a gift,” but this is in fact a donor illusion -donations support Heifer International’s general “agricultural assistance” activities. We have concerns about this general area and concerns about giving livestock specifically. Neither Heifer’s website nor its grant application have provided the kind of information needed to address these concerns.

  Full set of blog posts on Heifer International

• Smile Train: Prof. Steven Levitt of Freakonomics sees it as using an innovative model to deliver cleft surgeries for $250 apiece. Our analysis suggests that it has “more money than doctors” and that it is therefore out of room for more funding. Smile Train’s recent removal of the charts we questioned and its own explanation of why it is giving grants to other major charities appear to support our analysis. In addition, we have major concerns about surgery charities that Smile Train has not provided enough information to address.

  Full set of blog posts on Smile Train

• UNICEF offers donors the chance to “buy” vaccines and other items. We suspect that this offer is a donor illusion, but the larger issue is that UNICEF provides almost no information at all about its huge variety of activities and how they are going. Most giant international aid charities (Save the Children, World Vision, etc.) are similarly complex and opaque, with CARE coming the closest to meaningful transparency.

  Full set of blog posts on UNICEF

• The Acumen Fund stresses its metrics-driven approach to “patient capital,” investing in businesses that aim for long-term sustainability (i.e., profitability) and social impact. As of now, we see no substantive available information about its track record.

  Full discussion

• The Robin Hood Foundation also stresses its emphasis on metrics, quantification, and analytical thinking. But we have found that it is unwilling to share any information about the results of its grantmaking. (Note that it also pushes the “0% overhead” donor illusion.)

  Full set of blog posts on The Robin Hood Foundation

• The Millennium Villages project states that it is “demonstrating that it’s possible to escape … extreme poverty” (front page of its website). As far as we can tell, very few results have been demonstrated to date, with a few boosts in crop yields as its main evidence of impact - evidence we feel is highly questionable.

  Full discussion

• The Worldwide Fistula Fund has been praised by Nicholas Kristof as “giving new life to the pariahs.” Our assessment is that the Fund is pushing a debatable agenda whose effectiveness has yet to be shown.

  Full discussion

• The Carter Center is, in our view, by far the “best bet” (for an individual donor, our focus) of the charities listed in this post. It has achieved substantial, demonstrable impact in the past through its work on guinea worm eradication, and unlike all of the charities listed above, it publicly discloses substantive technical information on the impact of its past activities.

  However, there is a crucial piece of information missing: how it allocates its budget between programs (in particular, how much it spends on its stronger and better-documented vs. more questionable programs). It has declined to share this (basic, in our view) information in repeated exchanges.

  Full set of blog posts on The Carter Center

We think the above charities are fairly representative of “average” charities in international aid. Some tell better stories than others and some have more disclosure than others. But in almost all cases, international aid charities are (a) carrying out complex projects that can fail to do good (or even do harm) in a variety of ways, and (b) not systematically sharing the information that would make it possible to assess how their work is going.

GiveWell is devoted to finding charities in which we can have more confidence. We’ll be discussing our two top-rated charities working internationally in forthcoming posts.

It seems particularly hard to find information about the past impact of “gifts of livestock” programs (such as those promoted by Heifer International). I’ve been thinking about such programs conceptually, though, and I have a lot of trouble understanding the reasoning behind these programs. Two key points:

• It seems like giving out livestock brings with it all of the problems and challenges of giving out cash.
• It seems like giving out livestock also brings additional problems and challenges that don’t apply to giving out cash.

Giving out livestock brings the same problems and challenges as giving out cash

We’ve written before about the idea of cash transfers. One potential problem with giving out cash is that the more powerful people in a community may end up dominating (monopolizing?) the benefits. Giving out truly valuable gifts could interfere with power dynamics, incite jealousy, and fail to reach those that donors actually intend to help.

It seems to me that all of these concerns apply in full force to gifts of livestock. I doubt there are many people in the world who would turn down a free cow (I’m not sure I would). Even if one has no ability to take care of the cow, there is always the option of selling it (if one has access to markets) or simply slaughtering it and eating or selling the meat.

I couldn’t feel confident in a charity giving out livestock unless I saw compelling evidence that they were getting livestock to people in need, and not just to anyone (or just to the most powerful people) interested in free livestock.

Giving out livestock brings other problems and challenges as well

• Are the livestock in good health? Will they meet recipients’ expectations, or will they die or underproduce, potentially causing people to make bad plans and investments?
• Do the recipients of livestock gifts have the ability, in terms of knowledge and resources, to take care of the livestock well? (Similar problems as in the above bullet point could arise if they don’t.)
• Do the recipients of livestock intend to take care of the livestock well? Or is there reason to be concerned that gifts of livestock could lead to cruelty to animals?
• Are there other unforeseen consequences of introducing large numbers of livestock into a community? A few years ago, there were some allegations that “over grazing by goats in arid environments has disastrous effects on the fertility of the land … these ‘gifts’ merely add to the problems of hard-pressed communities because of the drain on limited resources the animal represents.” I haven’t been able to find any facts behind these allegations and I’m not sure what they’re based on.
• Most importantly, might recipients benefit more from other gifts - and why shouldn’t they make that assessment themselves? Perhaps the story Heifer International tells is correct, and livestock would make a tremendous difference for a family. If that’s the case, then a cash gift could be expected to be spent on livestock. If one of the many concerns above applies - or livestock is not what’s most helpful for any number of reasons we simply haven’t thought of - then a cash gift will be used on something else.

  Why is it better for a charity to decide people’s needs for them? This question isn’t entirely rhetorical - there could be a good reason - but it seems that the burden of proof on a statement like “A cow is better for you than anything else you could buy with what the cow costs” should be on the charity.

Bottom line

I have trouble understanding the idea of livestock gifts, from the perspective of maximizing positive impact.

I understand that they make a good ad campaign, possibly because they draw people’s attention to the possibility of using a one-time gift to permanently escape from poverty (even though a cash gift can just as easily lead to a story like this, and could also lead to a lot of other positive stories that we simply haven’t considered).

It’s a scary thought, but it seems possible to me that these programs exist entirely because of how they can be marketed to donors, instead of for any reasons relating to maximizing good accomplished. What am I missing?

The New York Times recently profiled Chess in the Schools:

The Chess-in-the-Schools program has sought to foster analytical skills on the theory that these will help students succeed academically. The group teaches 20,000 children a year and calculates that it has taught 425,000 children since 1986. Children gather to learn the game at the group’s headquarters in Manhattan.

It seems like 20 years and 425,000 children is quite a lot of investment in the “theory that [chess] will help students succeed academically.” The Times feature provides a calming justification for the investment: “Chess helps promote intellectual growth and has been shown to improve academic performance.” Let’s look at the evidence for this claim.

The study we found

An early-1990s study looks at achievement test scores of chess-playing students over two years at District 9 in the Bronx. It observes that (a) the overall average reading score improved among chessplayers by about 5 percentile points, but didn’t improve among the set of remaining District 9 students; (b) 15 of 22 second-year participants improved their reading scores by some amount, while only 491 of 1118 non-participants in the district - and 245 of 655 non-participants with high reading scores, improved.

This study is riddled with major problems:

• The numbers the researchers choose to compare seem arbitrary and possibly cherry-picked. Why do the researchers look at the “percentage who improved” among second-year chessplayers but not for both years? Why do they compare the second-year students to “high-performing nonparticipants,” but not give the same comparison when looking at all students?
• The problem of selection bias is unusually obvious here. They’re comparing kids who volunteered to play chess against those who didn’t. Think of the chess club members at your school, and ask yourself if they would have been just like all the other kids had chess club not been offered. There’s no reason to think these two groups of kids are otherwise similar or would be expected to respond similarly to school.
• This is a study of somewhere between 22 and 53 students at a single district in the early 1990s. Even if the study were highly rigorous, it would still be a long way from “proof that chess helps promote intellectual growth.”

The studies we couldn’t find

The Chess-in-the-Schools website states:

In 1991 and 1996, Stuart M. Margulies, Ph.D., a noted educational psychologist, conducted two studies examining the effects of chess on children’s reading scores. The studies demonstrated that students who participated in the chess program showed improved scores on standardized tests. The gains were even greater among children with low or average initial scores. Children who were in the non-chess playing control group showed no gains.

Another study in 1999, measured the impact of chess on the emotional intelligence of fifth graders. The results of the study were striking. The overall success rate in handling real life situations with emotional intelligence was 91.4% for the children who participated in the Chess-in-the-Schools program. In contrast, those who were not involved with the chess program had an average overall success rate of only 64.4%.

We’re guessing that the study we’re looking at is an update of the 1991 study since it references no previous studies and discusses results from 1991 and 1992. We can’t find the other studies anywhere. Chess-in-the-Schools provides neither links nor citations.

Even in the best-case scenario, it’s apparently been at least a decade since the last test of the Chess-in-the-Schools model.

“Chess helps promote intellectual growth and has been shown to improve academic performance?”

In researching charities, one of the more discouraging things we’ve learned is how little support it takes for a statement like “Chess helps promote intellectual growth and has been shown to improve academic performance” to be repeated by charities, donors, and even the media.

As far as we can tell, Chess-in-the-Schools is not a demonstrated success story. It’s just been promoted and scaled up like one.

I recently attended a seminar with the fascinating Seth Godin and heard an interesting anecdote about VisionSpring:

I could see that every single person who came to this meeting had enough money [$3] … to buy a pair of new reading glasses. And I could tell from how old they were that they were qualified and I knew what they did for a living so I knew that this would pay for itself in two weeks, three weeks, certainly in 20 years it’s going have a huge return on investment. So they get the demonstration, they take the eye test, they see that they need glasses, they take the sample glasses off, they walk over to the table where there are 8 kinds of glasses to choose from … all carefully wrapped in plastic … and 40% of the people bought a pair of glasses. 60% of the people left. And I’ve thought about this about a million times … If they knew how great the glasses would be, if they could overcome the momentum they had and the desire to keep the money … there’s no question they would have bought a pair of glasses. Maybe for $6 or $9 or $12, they could afford it, they needed it, they were in the right place at the right time, and yet the transfer didn’t occur.

To Mr. Godin, it seemed obvious that the customers should be buying glasses, and that they were being held back by “momentum.” He proposed a change:

Instead of giving the person the eye test, taking off the glasses and having them go over … and now make the decision do you care enough about yourself to buy those, instead, give the guy the eye test, and say those are your glasses, you owe me $3. Now the person has to make a new decision, which is better, giving up the $3 and keeping on what I have or going to the trouble of taking them off and reminding myself that I don’t deserve to see? And when you do that it turns out that the close rate goes up 30%.

To be sure, it sounds like an impressive improvement for a simple change. And yet a 30% improvement on a 40% close rate still leaves about half the people not spending the glasses - now under circumstances that arguably make it pretty difficult to turn them down.

Is it possible that Mr. Godin – and the charity he was observing – have simply overestimated the demand for what they’re doing? Underestimated the extent to which imperfect substitutes may be available? It appears that Village Phone did exactly this in at least one case.

It’s easy to be sure that your product is great and that it’s needed. Yet if you’re wrong, and you have donors subsidize it, you may essentially be giving out cash in a less efficient, less empowering way.

Mr. Godin analogized the situation to fundraising:

You’re sitting in class and the person next to you … is coughing … and you take out a container of Fisherman’s Friend menthol lozenges and you offer it to her and she puts it in her mouth and the coughing goes away and everyone is happy … and yet you’ve been on those calls to raise money from a donor, who apparently has money to donate … and you hand them the equivalent of a Fisherman’s Friend, “look at this program, we’ve been working really hard on it, it’s really important, we need your money” and they say “I need to get back to you” and you know what that means …

To me, there’s a major problem with the assumption that a charity’s program is at effective at solving complex social problems as a Fisherman’s Friend is at soothing throats.

Confidence in your product is great, but it can be misplaced. When charities let assumptions like “Everyone here needs glasses” and “Our program is reliable as a cough drop” go unexamined, they’re going to be left scratching their heads at results-oriented donors.

Ever since I heard about the Grameen Foundation’s Village Phone program, I’ve been optimistic. The program involves helping people in remote villages run pay-for-use cellphone services: they get their cellphone, and a loan to buy it, via Grameen, then charge other villagers to use it. It’s an approach to fighting poverty that’s (a) relatively new; (b) using a product that hasn’t been available for a long time but seems clearly useful to anyone doing business in remote areas; (c) utilizing a “franchise model” where people in the villages take a stake in the product.

It was near the top of my “Probably helping people, even though we don’t yet have systematic evaluation yet” list. Now Chris Blattman points to a discouraging evaluation that found “absolutely no impact of the phones on trading activity or availability of goods in local markets” and very small (non-significant) impacts on profits and measures of well-being (school enrollment, consumption of meat, etc.)

This bottom-line result does not, by itself, mean the program “doesn’t work.” It could work very differently in different contexts (discussed below), and there are some possible issues with the paper (which is very recent, and is not a randomized controlled trial). But one thing I like about the study is that it doesn’t just discuss impact - it examines many aspects of the program, and exposes assumptions that may otherwise have gone unquestioned.

Assumption 1: the phones are in high demand and operators easily cover their costs. In fact, usage of the phones was around 4 hours a month, or 8 minutes a day (pg 19). As a result, profits from the phones were not enough to keep up with loan payments (pg 19). Grameen reportedly has responded by changing loan and franchise terms (pg 30). Tuvugane (pg 5), a less sophisticated phone product that was already common in the villages, may have been good enough for most purposes.

Assumption 2: farmers who use the phones benefit from better pricing power. Even though farmers with access to the phones became much more likely to arrange their own transport to market, there was no apparent effect on the prices they received for their goods, possibly due to established relationships with buyers (pg 16).

Assumption 3: if someone chooses to become a cellphone operator, they’re going to benefit from it. In fact, there was a very strange pattern in the businesses of people who became phone operators. Their hours worked rose significantly both for their new phone business and for their already-existing businesses, but their profits and wages paid did not rise (pgs 17-18). A possible explanation is that operators wanted to be available for cellphone users and so stayed at their workplaces longer, but that the extra hours didn’t translate into extra profits. In any case, it’s a pattern that doesn’t seem encouraging, and seems to deserve further investigation.

Bottom line: a product that was supposed to be helpful and in high demand arguably ended up as a bad investment for the franchise operators. This doesn’t mean it shouldn’t have been tried, or that it shouldn’t be tried in the future. But it points to the importance of testing assumptions empirically, rather than scaling up a program as widely as possible based on an appealing story.

What’s wrong with this “evidence of impact” for high-profile charities?

• The Millennium Villages project’s main evidence of impact so far appears to be increasing crop yields by a factor of 4-15x in 3 villages. (H/T to the recent coverage of Millennium Villages on Aid Watch)
• A KickStart evaluation (available on our website (DOC)) reports that “Family/household incomes of 3,000 farmers increased by 100% within 24 months following acquisition of micro-irrigation pumps.”
• The Carter Center’s agriculture program “provides the tools and means to help farmers in sub-Saharan Africa increase their crop yields through agricultural development, sometimes two- or even threefold.”

Among other possible problems, two major issues jump out:

1. No context on what “normal” variation in incomes looks like for poor farmers. Some years have more favorable weather - and local economic situations - than others. Enough that one year’s income or crop yield could be double another’s? 4x? 20x?

Unfortunately, one of the better pieces of “evidence” that jumps to mind is a 75-year-old novel, The Good Earth, whose farmer protagonist is comfortable one year and has literally zero income the next, for no other reason than the weather. If a given year’s yield were close enough to zero, the next year could be a huge increase (2x, 4x, 20x or more) simply by returning to normal.

I have seen little information on the local year-to-year volatility that poor farmers can experience, but I imagine that it (a) varies greatly from region to region and (b) could easily involve incomes falling and jumping by enormous amounts.

None of the above reports provide any context on this question, beyond qualitative statements about how favorable the rains were in each year examined. None of them employ any sort of “comparison group” of farmers (aside from one vague reference to “farms not using improved seeds and fertilizers” in the Malawi Millennium Village). Ultimately, none accomplish one of the most basic goals of an evaluation: giving a sense of how likely the “gains” they describe are to have arisen by pure chance.

With larger sample sizes, we might be able to use country-level volatility for context. But that brings me to the next problem.

2. We have no assurance that the described gains are representative, as opposed to “cherry-picked.”

All of the above organizations have reputations for consistent and thorough monitoring and evaluation, yet in all cases, we find ourselves looking for “impact” from a tiny subset of their projects.

• The Millennium Villages evaluation “includes results from the Sauri baseline survey, annual reports from Sauri and Koraro and crop-harvest data from Malawi.” How were these three chosen from among the 78 villages mentioned in the abstract?* The report doesn’t say.
• KickStart implies that it is constantly and consistently tracking impact, yet the report linked above discusses a single project over a single year.
• The Carter Center is somewhat explicit that its number is a “high-end” one (”sometimes two- or even threefold”). Examining the most recent annual report for its agriculture program shows a very small upward trend, if any, in crop yields across the countries in which it is working.

Some ways to produce more compelling evidence of impact

1. Be clear about what is being measured and what is being published, and when. It seems to us that in this area, charity evaluation lags far behind clinical trials, which are constantly registered before they are complete so people can track their progress. (The Poverty Action Lab is similarly transparent with its own ongoing projects.)
2. More sample size; more context; use of comparison groups. Discussed above.
3. Look for more sustained improvements in people’s lives. One measure I find superior to straight “income” or “crop yields” is asset accumulation. A jump in income could be temporary; if someone upgrades their roof or sanitation, it’s likely that at least they expect the gain to be a real and lasting one. The Village Enterprise Fund’s evaluation is one of the better charity evaluations I’ve seen in the area of economic empowerment, partly because it focuses on standard of living rather than a simple measure of income.

*It’s possible that the yields mentioned are for “clusters” of villages rather than individual villages; there are only 12 clusters. However, the source documents available for Sauri and Koraro appear to be at the village rather than the cluster level, and the details of how the measurements were made are unclear.