In a previous post, we asked whether Haiti earthquake relief has “room for more funding” and concluded that, in general, it isn’t clear. (For more on the general topic of “room for more funding,” see our 5-post series on the topic.)

Of course, it probably makes a big difference which organization we’re talking about. We’ve seen a lot of different charities soliciting funds in the context of the Haiti earthquake, and some of them may have greater abilities than others to translate funding into relief assistance. The problem for donors is that by and large, these charities aren’t accompanying their appeals with the information we’d need to have a sense of their “room for more funding.”

The key questions we feel charities should be answering

We assert that any organization raising funds in the context of Haiti earthquake relief should provide clear, prominent answers to the following questions:

1. How much are you trying to raise?
2. Roughly speaking, what activities are you seeking to fund?
3. How much have you raised so far?
4. If you raise more than your target, what will you do with the remaining funds?

Answers to such questions would make it possible to hold organizations accountable (well after the fact) for how they actually spent money raised in their relief appeals.

Charities’ answers to these questions

We’ve examined the online appeals of charities that (a) have raised $10 million or more in the context of Haiti relief, according to the Chronicle of Philanthropy’s tally or (b) have come up via Google Adwords on relevant searches (”Haiti”, “Haiti relief”, “Haiti donations”, etc.) (We also included Yele Haiti due to the attention it’s received, even though it did not meet either of these criteria.) It seems fair to say that all of these charities have done significant fundraising in the context of the Haiti relief effort.

For each, we looked on their website for answers to the above questions, both by clicking relevant links and by Google-searching the site. (Note that this work was mostly done last week.) We provide our results in Excel format.

Most (not all charities) provide at least an overview of what sort of aid they are looking to provide. However, only 6 out of 17 appear to give any target for how much they’re looking to raise; only three appear to be sharing how much they’ve raised to date (although many more appear to be sharing this information via the Chronicle of Philanthropy’s tally); and only two (the two mentioned in our previous post, Heifer International and Doctors Without Borders) are explicit about what they will do with “extra” funds.

We don’t believe that a top-down pooling of funds is the only way to make sure money gets to the organizations that can use it productively. We see potential in the idea of different charities’ raising funds for different efforts, as long as they make it clear how much they are looking for and why; make it possible to hold them accountable for how they spend the funds they raise; and are explicit about the point at which they’d spend funds on other potentially worthy causes. However, from what we can see, charities aren’t currently sharing enough information along these lines.

Normally, we focus on identifying outstanding charities, and minimize the time spent on opaque or otherwise lackluster ones. But lately, we’ve gone into a bit more detail about our take on several of the best-known and most appealing charities out there.

What all of the charities below have in common is that (a) we have major questions and concerns about their activities; (b) the information necessary to see how serious these concerns are does not seem to be available. (In most cases our assessment is based on significant back-and-forth with the charities themselves, though in some cases we are going off their website.)

• Kiva: commonly perceived as a way to (a) make interest-free loans to (b) entrepreneurs you personally select. In fact, it is neither. (See GiveWell Board member Tim Ogden’s summary of the recent discussion/controversy.)

  We ultimately see a Kiva “loan” as a gift to a microfinance institution, and we have major concerns about supporting any but the most outstanding microfinance institutions as well as concerns about Kiva’s due diligence process specifically.

  Full set of blog posts on Kiva

• Grameen Foundation and other large U.S. microfinance charities: in addition to our general concerns about microfinance charities (and the fact that these charities have not provided the information that would address them), we find that these organizations are rarely even clear about what their value-added is. Also see the disappointing (and eye-opening) evaluation of the Village Phone program.

  Full set of blog posts on Grameen Foundation

• Heifer International: commonly perceived as a way to “give a cow to a poor family as a gift,” but this is in fact a donor illusion -donations support Heifer International’s general “agricultural assistance” activities. We have concerns about this general area and concerns about giving livestock specifically. Neither Heifer’s website nor its grant application have provided the kind of information needed to address these concerns.

  Full set of blog posts on Heifer International

• Smile Train: Prof. Steven Levitt of Freakonomics sees it as using an innovative model to deliver cleft surgeries for $250 apiece. Our analysis suggests that it has “more money than doctors” and that it is therefore out of room for more funding. Smile Train’s recent removal of the charts we questioned and its own explanation of why it is giving grants to other major charities appear to support our analysis. In addition, we have major concerns about surgery charities that Smile Train has not provided enough information to address.

  Full set of blog posts on Smile Train

• UNICEF offers donors the chance to “buy” vaccines and other items. We suspect that this offer is a donor illusion, but the larger issue is that UNICEF provides almost no information at all about its huge variety of activities and how they are going. Most giant international aid charities (Save the Children, World Vision, etc.) are similarly complex and opaque, with CARE coming the closest to meaningful transparency.

  Full set of blog posts on UNICEF

• The Acumen Fund stresses its metrics-driven approach to “patient capital,” investing in businesses that aim for long-term sustainability (i.e., profitability) and social impact. As of now, we see no substantive available information about its track record.

  Full discussion

• The Robin Hood Foundation also stresses its emphasis on metrics, quantification, and analytical thinking. But we have found that it is unwilling to share any information about the results of its grantmaking. (Note that it also pushes the “0% overhead” donor illusion.)

  Full set of blog posts on The Robin Hood Foundation

• The Millennium Villages project states that it is “demonstrating that it’s possible to escape … extreme poverty” (front page of its website). As far as we can tell, very few results have been demonstrated to date, with a few boosts in crop yields as its main evidence of impact - evidence we feel is highly questionable.

  Full discussion

• The Worldwide Fistula Fund has been praised by Nicholas Kristof as “giving new life to the pariahs.” Our assessment is that the Fund is pushing a debatable agenda whose effectiveness has yet to be shown.

  Full discussion

• The Carter Center is, in our view, by far the “best bet” (for an individual donor, our focus) of the charities listed in this post. It has achieved substantial, demonstrable impact in the past through its work on guinea worm eradication, and unlike all of the charities listed above, it publicly discloses substantive technical information on the impact of its past activities.

  However, there is a crucial piece of information missing: how it allocates its budget between programs (in particular, how much it spends on its stronger and better-documented vs. more questionable programs). It has declined to share this (basic, in our view) information in repeated exchanges.

  Full set of blog posts on The Carter Center

We think the above charities are fairly representative of “average” charities in international aid. Some tell better stories than others and some have more disclosure than others. But in almost all cases, international aid charities are (a) carrying out complex projects that can fail to do good (or even do harm) in a variety of ways, and (b) not systematically sharing the information that would make it possible to assess how their work is going.

GiveWell is devoted to finding charities in which we can have more confidence. We’ll be discussing our two top-rated charities working internationally in forthcoming posts.

Early in 2009, we were extremely excited about The Carter Center. It seemed so strong that we devoted weeks to understanding it in depth.

As discussed in a blog post we made at the time, several of its programs work on extremely promising “neglected tropical disease control” activities, and there’s a truly unusual amount of disclosure from these programs. It appeared that the Carter Center is near the top of the heap both for what it’s doing and for how it’s sharing information. To boot, it was directly involved in one of the most cited global health success stories, the near-eradication of guinea worm.

The Carter Center also has several programs that don’t seem as promising. At first we nearly dismissed/overlooked these programs. But as we dug deeper, we realized that just because a charity emphasizes its best programs doesn’t mean it’s spending most of its funds on them. Oddly, the one piece of information we couldn’t seem to find anywhere on its website was how much of its budget was allocated to each program. The back-of-the-envelope calculations we did surprised us: the heavily documented river blindness program seemed as though it must be tiny, while the agriculture program hadn’t published anything since 2005 but appeared at that time to be taking up around 10% of the total budget.

We got in touch with The Carter Center and asked for a budget breakdown by program. We spoke to a senior representative and followed up with him 4 times. We even tried getting a connection of ours who has been a major Carter Center donor to ask for the information. It kept getting put off. Today we still don’t have this information.

To be honest, at this point we don’t know whether the “flagship” disease-control programs are at the core of the Carter Center’s work or act as more of a “hook” for donors while it focuses on things like fellowships for mental health journalism. And we have no sense of what a donor accomplishes by giving them a small gift (a gift that, however it’s officially designated, is likely effectively going to fund what the Carter Center wants it to fund due to the issue of fungibility).

To give a sense of the variety of program type and quality, here’s where we stand on a few select programs:

• Guinea worm eradication: as mentioned above, progress to date has been impressive. But the case for further funding is murky - the Carter Center proposes $10-15 million per year to fight the last vestiges of a disease that now affects very few people.
• Lymphatic filariasis control program: cheaply treating a horrible and neglected disease, this program is our favorite of the bunch, but we have no idea how large it is or whether more donations would result in expansion of it.
• Agriculture program: appears to be trying to bring the Green Revolution to Africa, but the last update was in 2005 and did not show encouraging results. The program’s “Staff” page appears only to list Dr. Norman Borlaug, whose accomplishments are truly astounding but who passed away earlier this year.
• Ethiopian Public Health Training Initiative: we’re very much in favor of training more public health professionals if it can be done capably. But this program is very thinly documented, in contrast to the direct disease control programs. Are participants in the program being assessed afterward to see if they’ve improved their skills? Are they contributing to Ethiopia’s health care system?

We wish the Carter Center were as transparent about its budget as it is about (some of) its program activities.

What’s wrong with this “evidence of impact” for high-profile charities?

• The Millennium Villages project’s main evidence of impact so far appears to be increasing crop yields by a factor of 4-15x in 3 villages. (H/T to the recent coverage of Millennium Villages on Aid Watch)
• A KickStart evaluation (available on our website (DOC)) reports that “Family/household incomes of 3,000 farmers increased by 100% within 24 months following acquisition of micro-irrigation pumps.”
• The Carter Center’s agriculture program “provides the tools and means to help farmers in sub-Saharan Africa increase their crop yields through agricultural development, sometimes two- or even threefold.”

Among other possible problems, two major issues jump out:

1. No context on what “normal” variation in incomes looks like for poor farmers. Some years have more favorable weather - and local economic situations - than others. Enough that one year’s income or crop yield could be double another’s? 4x? 20x?

Unfortunately, one of the better pieces of “evidence” that jumps to mind is a 75-year-old novel, The Good Earth, whose farmer protagonist is comfortable one year and has literally zero income the next, for no other reason than the weather. If a given year’s yield were close enough to zero, the next year could be a huge increase (2x, 4x, 20x or more) simply by returning to normal.

I have seen little information on the local year-to-year volatility that poor farmers can experience, but I imagine that it (a) varies greatly from region to region and (b) could easily involve incomes falling and jumping by enormous amounts.

None of the above reports provide any context on this question, beyond qualitative statements about how favorable the rains were in each year examined. None of them employ any sort of “comparison group” of farmers (aside from one vague reference to “farms not using improved seeds and fertilizers” in the Malawi Millennium Village). Ultimately, none accomplish one of the most basic goals of an evaluation: giving a sense of how likely the “gains” they describe are to have arisen by pure chance.

With larger sample sizes, we might be able to use country-level volatility for context. But that brings me to the next problem.

2. We have no assurance that the described gains are representative, as opposed to “cherry-picked.”

All of the above organizations have reputations for consistent and thorough monitoring and evaluation, yet in all cases, we find ourselves looking for “impact” from a tiny subset of their projects.

• The Millennium Villages evaluation “includes results from the Sauri baseline survey, annual reports from Sauri and Koraro and crop-harvest data from Malawi.” How were these three chosen from among the 78 villages mentioned in the abstract?* The report doesn’t say.
• KickStart implies that it is constantly and consistently tracking impact, yet the report linked above discusses a single project over a single year.
• The Carter Center is somewhat explicit that its number is a “high-end” one (”sometimes two- or even threefold”). Examining the most recent annual report for its agriculture program shows a very small upward trend, if any, in crop yields across the countries in which it is working.

Some ways to produce more compelling evidence of impact

1. Be clear about what is being measured and what is being published, and when. It seems to us that in this area, charity evaluation lags far behind clinical trials, which are constantly registered before they are complete so people can track their progress. (The Poverty Action Lab is similarly transparent with its own ongoing projects.)
2. More sample size; more context; use of comparison groups. Discussed above.
3. Look for more sustained improvements in people’s lives. One measure I find superior to straight “income” or “crop yields” is asset accumulation. A jump in income could be temporary; if someone upgrades their roof or sanitation, it’s likely that at least they expect the gain to be a real and lasting one. The Village Enterprise Fund’s evaluation is one of the better charity evaluations I’ve seen in the area of economic empowerment, partly because it focuses on standard of living rather than a simple measure of income.

*It’s possible that the yields mentioned are for “clusters” of villages rather than individual villages; there are only 12 clusters. However, the source documents available for Sauri and Koraro appear to be at the village rather than the cluster level, and the details of how the measurements were made are unclear.

In Part I, I laid out my case for The Carter Center, and why we think they’re worth investigating deeply. I also said that we’re not yet ready to recommend them because we have some unanswered questions. Here they are:

Relevance of your donation

When I give, I want to know that the organization needs my support and is going to accomplish more good (than they otherwise would have) because of my support. Because of The Carter Center’s strong track record, I wonder whether they can already effectively raise all the money they want. That is, I wonder whether money isn’t the bottleneck to serving more people. That question is supported by looking at Carter’s current financial situation. Revenues have risen roughly in-line with expenses over the past few years (see chart 1 below), maintaining their (relatively large) assets:expense ratio of 3:1 (see chart 2, below). All data comes from Carter Center annual reports, available on their website here.

Less-proven and less-monitored programs.

Many of The Carter Center’s programs have extremely strong independent evidence supporting them (e.g., their guinea worm, river blindness, lymphatic filariasis, and schistosomiasis programs). These programs also have the type of strong, consistent monitoring I mentioned in the previous post. However, the evidence case for some of their other programs is weaker (in particular their trachoma, agriculture, and public health training initiative), and the monitoring provided for those programs is both a) less consistent and b) less compelling than the monitoring provided for the others.

Running some less-proven projects doesn’t necessarily mean we won’t recommend them, especially if those programs are relatively small, but:

Lack of financial transparency

The Carter Center is one of the (if not the) most transparent organizations I’ve ever come across when it comes to monitoring their activities. Carter not only lays out what they do in extreme detail — for the intrepid, the most recent river blindness program report (PDF) is representative of Carter Center monitoring — they discuss program trade-offs they make, lessons they’ve learned, and potential future obstacles. But, they don’t publish any cost data on their website, other than the very broad and general 990 data and some sporadic program-specific information. To properly evaluate The Carter Center, we need to know the portion of the budget that’s allocated to any less-proven programs. In addition, we want tell you what your donation accomplishes to The Carter Center. Does $1,000 prevent 1 case of blindness? 2? 10? We need better information on costs to make that call.

Those are the questions we’re working on answering now. Whatever we find, you’ll get our answers soon.

I’ve spent a good part of the past month reviewing The Carter Center in depth. We found out about The Carter Center though referrals from an advisor and donor, and through the Gates Award for Global Health.

We chose to investigate The Carter Center further for four reasons:

1. Strong track record of success. The Carter Center has been credited with leading the global effort to eradicate guinea worm. In 1986, when Carter got involved, there were 3.5m annual cases of guinea worm. In 2008, there were fewer than 5,000. Guinea worm has been eliminated from 12 of the 18 countries that were once endemic. This chart shows the decline in cases from 1989-2007 (PDF). (More information about guinea worm and its symptoms is available on The Carter Center website.)
2. Strong program selection. Other Carter Center programs have strong evidence of effectiveness supporting them. The interventions they implement to control or eliminate neglected tropical disease including river blindness, trachoma, lymphatic filariasis, and schistosomiasis are recommended by the experts at the Disease Control Priorities Project.
3. Monitoring and evaluation. For its health programs, The Carter Center often monitors not only the number of drugs distributed in each region but disease prevalence directly. This monitoring isn’t evident for every single program or region, but it is far more consistent than any other complex organization we’ve seen before.
4. Transparency. All the information we’ve used to research The Carter Center is available publicly on their website. For example, the detailed program reports I mentioned above are available here. For an example of the type of monitoring conducted for its guinea worm program, see the latest monthly report linked here.

I still have some large, unanswered questions about The Carter Center, and we aren’t yet ready to recommend them.