Normally, we focus on identifying outstanding charities, and minimize the time spent on opaque or otherwise lackluster ones. But lately, we’ve gone into a bit more detail about our take on several of the best-known and most appealing charities out there.

What all of the charities below have in common is that (a) we have major questions and concerns about their activities; (b) the information necessary to see how serious these concerns are does not seem to be available. (In most cases our assessment is based on significant back-and-forth with the charities themselves, though in some cases we are going off their website.)

• Kiva: commonly perceived as a way to (a) make interest-free loans to (b) entrepreneurs you personally select. In fact, it is neither. (See GiveWell Board member Tim Ogden’s summary of the recent discussion/controversy.)

  We ultimately see a Kiva “loan” as a gift to a microfinance institution, and we have major concerns about supporting any but the most outstanding microfinance institutions as well as concerns about Kiva’s due diligence process specifically.

  Full set of blog posts on Kiva

• Grameen Foundation and other large U.S. microfinance charities: in addition to our general concerns about microfinance charities (and the fact that these charities have not provided the information that would address them), we find that these organizations are rarely even clear about what their value-added is. Also see the disappointing (and eye-opening) evaluation of the Village Phone program.

  Full set of blog posts on Grameen Foundation

• Heifer International: commonly perceived as a way to “give a cow to a poor family as a gift,” but this is in fact a donor illusion -donations support Heifer International’s general “agricultural assistance” activities. We have concerns about this general area and concerns about giving livestock specifically. Neither Heifer’s website nor its grant application have provided the kind of information needed to address these concerns.

  Full set of blog posts on Heifer International

• Smile Train: Prof. Steven Levitt of Freakonomics sees it as using an innovative model to deliver cleft surgeries for $250 apiece. Our analysis suggests that it has “more money than doctors” and that it is therefore out of room for more funding. Smile Train’s recent removal of the charts we questioned and its own explanation of why it is giving grants to other major charities appear to support our analysis. In addition, we have major concerns about surgery charities that Smile Train has not provided enough information to address.

  Full set of blog posts on Smile Train

• UNICEF offers donors the chance to “buy” vaccines and other items. We suspect that this offer is a donor illusion, but the larger issue is that UNICEF provides almost no information at all about its huge variety of activities and how they are going. Most giant international aid charities (Save the Children, World Vision, etc.) are similarly complex and opaque, with CARE coming the closest to meaningful transparency.

  Full set of blog posts on UNICEF

• The Acumen Fund stresses its metrics-driven approach to “patient capital,” investing in businesses that aim for long-term sustainability (i.e., profitability) and social impact. As of now, we see no substantive available information about its track record.

  Full discussion

• The Robin Hood Foundation also stresses its emphasis on metrics, quantification, and analytical thinking. But we have found that it is unwilling to share any information about the results of its grantmaking. (Note that it also pushes the “0% overhead” donor illusion.)

  Full set of blog posts on The Robin Hood Foundation

• The Millennium Villages project states that it is “demonstrating that it’s possible to escape … extreme poverty” (front page of its website). As far as we can tell, very few results have been demonstrated to date, with a few boosts in crop yields as its main evidence of impact - evidence we feel is highly questionable.

  Full discussion

• The Worldwide Fistula Fund has been praised by Nicholas Kristof as “giving new life to the pariahs.” Our assessment is that the Fund is pushing a debatable agenda whose effectiveness has yet to be shown.

  Full discussion

• The Carter Center is, in our view, by far the “best bet” (for an individual donor, our focus) of the charities listed in this post. It has achieved substantial, demonstrable impact in the past through its work on guinea worm eradication, and unlike all of the charities listed above, it publicly discloses substantive technical information on the impact of its past activities.

  However, there is a crucial piece of information missing: how it allocates its budget between programs (in particular, how much it spends on its stronger and better-documented vs. more questionable programs). It has declined to share this (basic, in our view) information in repeated exchanges.

  Full set of blog posts on The Carter Center

We think the above charities are fairly representative of “average” charities in international aid. Some tell better stories than others and some have more disclosure than others. But in almost all cases, international aid charities are (a) carrying out complex projects that can fail to do good (or even do harm) in a variety of ways, and (b) not systematically sharing the information that would make it possible to assess how their work is going.

GiveWell is devoted to finding charities in which we can have more confidence. We’ll be discussing our two top-rated charities working internationally in forthcoming posts.

It seems particularly hard to find information about the past impact of “gifts of livestock” programs (such as those promoted by Heifer International). I’ve been thinking about such programs conceptually, though, and I have a lot of trouble understanding the reasoning behind these programs. Two key points:

• It seems like giving out livestock brings with it all of the problems and challenges of giving out cash.
• It seems like giving out livestock also brings additional problems and challenges that don’t apply to giving out cash.

Giving out livestock brings the same problems and challenges as giving out cash

We’ve written before about the idea of cash transfers. One potential problem with giving out cash is that the more powerful people in a community may end up dominating (monopolizing?) the benefits. Giving out truly valuable gifts could interfere with power dynamics, incite jealousy, and fail to reach those that donors actually intend to help.

It seems to me that all of these concerns apply in full force to gifts of livestock. I doubt there are many people in the world who would turn down a free cow (I’m not sure I would). Even if one has no ability to take care of the cow, there is always the option of selling it (if one has access to markets) or simply slaughtering it and eating or selling the meat.

I couldn’t feel confident in a charity giving out livestock unless I saw compelling evidence that they were getting livestock to people in need, and not just to anyone (or just to the most powerful people) interested in free livestock.

Giving out livestock brings other problems and challenges as well

• Are the livestock in good health? Will they meet recipients’ expectations, or will they die or underproduce, potentially causing people to make bad plans and investments?
• Do the recipients of livestock gifts have the ability, in terms of knowledge and resources, to take care of the livestock well? (Similar problems as in the above bullet point could arise if they don’t.)
• Do the recipients of livestock intend to take care of the livestock well? Or is there reason to be concerned that gifts of livestock could lead to cruelty to animals?
• Are there other unforeseen consequences of introducing large numbers of livestock into a community? A few years ago, there were some allegations that “over grazing by goats in arid environments has disastrous effects on the fertility of the land … these ‘gifts’ merely add to the problems of hard-pressed communities because of the drain on limited resources the animal represents.” I haven’t been able to find any facts behind these allegations and I’m not sure what they’re based on.
• Most importantly, might recipients benefit more from other gifts - and why shouldn’t they make that assessment themselves? Perhaps the story Heifer International tells is correct, and livestock would make a tremendous difference for a family. If that’s the case, then a cash gift could be expected to be spent on livestock. If one of the many concerns above applies - or livestock is not what’s most helpful for any number of reasons we simply haven’t thought of - then a cash gift will be used on something else.

  Why is it better for a charity to decide people’s needs for them? This question isn’t entirely rhetorical - there could be a good reason - but it seems that the burden of proof on a statement like “A cow is better for you than anything else you could buy with what the cow costs” should be on the charity.

Bottom line

I have trouble understanding the idea of livestock gifts, from the perspective of maximizing positive impact.

I understand that they make a good ad campaign, possibly because they draw people’s attention to the possibility of using a one-time gift to permanently escape from poverty (even though a cash gift can just as easily lead to a story like this, and could also lead to a lot of other positive stories that we simply haven’t considered).

It’s a scary thought, but it seems possible to me that these programs exist entirely because of how they can be marketed to donors, instead of for any reasons relating to maximizing good accomplished. What am I missing?

There’s an persistent conflict in international charity:

• It feels great to be able to say, “My donation helped THIS person.”
• But it’s rarely – if ever – practical for that sort of connection to be real.

As a result, international charities tend to create “donor illusions” by implying that donations can be attributed more tangibly, reliably and specifically than they really are. Some charities are more purposefully misleading than others, and some have more prominent and clear disclosures than others, but we feel that all of the cases below end up misleading many donors.

Kiva.org: making a loan
Illusion: You pick a developing-world entrepreneur who needs a loan. You lend money to that entrepreneur, interest-free. With you, s/he wouldn’t have gotten the loan.
Reality: The entrepreneur you’re viewing in your browser probably already got his/her loan and is probably paying significant interest on it. What you’re really doing is sending money to a microfinance institution that uses it as it sees fit (including for loans to less creditworthy people).

Details at GiveWell Board member Tim Ogden’s summary of the recent Kiva debate.

Child sponsorship: supporting a child

Illusion: through an organization such as Save the Children, your money supports a specific child.
Reality: as Save the Children now discloses, “Your sponsorship contributions are not given directly to a child. Instead, your contributions are pooled with those of other sponsors to provide community-based programming for all eligible children in the area.” See this David Roodman post (starting with “The Kiva Story”) for the interesting, scandal-ridden history of this practice.

Heifer International: giving livestock

Illusion: your donation pays for a cow for a specific developing-world family, helping it earn a better living.
Reality: as the fine print says, “Gifts made through this catalog represent a gift to the entire mission.” The entire mission generally includes a lot beyond livestock, including difficult projects like rural extension services.

Other donor illusions are more subtle. More on this in a future post.

What’s wrong with this “evidence of impact” for high-profile charities?

• The Millennium Villages project’s main evidence of impact so far appears to be increasing crop yields by a factor of 4-15x in 3 villages. (H/T to the recent coverage of Millennium Villages on Aid Watch)
• A KickStart evaluation (available on our website (DOC)) reports that “Family/household incomes of 3,000 farmers increased by 100% within 24 months following acquisition of micro-irrigation pumps.”
• The Carter Center’s agriculture program “provides the tools and means to help farmers in sub-Saharan Africa increase their crop yields through agricultural development, sometimes two- or even threefold.”

Among other possible problems, two major issues jump out:

1. No context on what “normal” variation in incomes looks like for poor farmers. Some years have more favorable weather - and local economic situations - than others. Enough that one year’s income or crop yield could be double another’s? 4x? 20x?

Unfortunately, one of the better pieces of “evidence” that jumps to mind is a 75-year-old novel, The Good Earth, whose farmer protagonist is comfortable one year and has literally zero income the next, for no other reason than the weather. If a given year’s yield were close enough to zero, the next year could be a huge increase (2x, 4x, 20x or more) simply by returning to normal.

I have seen little information on the local year-to-year volatility that poor farmers can experience, but I imagine that it (a) varies greatly from region to region and (b) could easily involve incomes falling and jumping by enormous amounts.

None of the above reports provide any context on this question, beyond qualitative statements about how favorable the rains were in each year examined. None of them employ any sort of “comparison group” of farmers (aside from one vague reference to “farms not using improved seeds and fertilizers” in the Malawi Millennium Village). Ultimately, none accomplish one of the most basic goals of an evaluation: giving a sense of how likely the “gains” they describe are to have arisen by pure chance.

With larger sample sizes, we might be able to use country-level volatility for context. But that brings me to the next problem.

2. We have no assurance that the described gains are representative, as opposed to “cherry-picked.”

All of the above organizations have reputations for consistent and thorough monitoring and evaluation, yet in all cases, we find ourselves looking for “impact” from a tiny subset of their projects.

• The Millennium Villages evaluation “includes results from the Sauri baseline survey, annual reports from Sauri and Koraro and crop-harvest data from Malawi.” How were these three chosen from among the 78 villages mentioned in the abstract?* The report doesn’t say.
• KickStart implies that it is constantly and consistently tracking impact, yet the report linked above discusses a single project over a single year.
• The Carter Center is somewhat explicit that its number is a “high-end” one (”sometimes two- or even threefold”). Examining the most recent annual report for its agriculture program shows a very small upward trend, if any, in crop yields across the countries in which it is working.

Some ways to produce more compelling evidence of impact

1. Be clear about what is being measured and what is being published, and when. It seems to us that in this area, charity evaluation lags far behind clinical trials, which are constantly registered before they are complete so people can track their progress. (The Poverty Action Lab is similarly transparent with its own ongoing projects.)
2. More sample size; more context; use of comparison groups. Discussed above.
3. Look for more sustained improvements in people’s lives. One measure I find superior to straight “income” or “crop yields” is asset accumulation. A jump in income could be temporary; if someone upgrades their roof or sanitation, it’s likely that at least they expect the gain to be a real and lasting one. The Village Enterprise Fund’s evaluation is one of the better charity evaluations I’ve seen in the area of economic empowerment, partly because it focuses on standard of living rather than a simple measure of income.

*It’s possible that the yields mentioned are for “clusters” of villages rather than individual villages; there are only 12 clusters. However, the source documents available for Sauri and Koraro appear to be at the village rather than the cluster level, and the details of how the measurements were made are unclear.