Normally, we focus on identifying outstanding charities, and minimize the time spent on opaque or otherwise lackluster ones. But lately, we’ve gone into a bit more detail about our take on several of the best-known and most appealing charities out there.

What all of the charities below have in common is that (a) we have major questions and concerns about their activities; (b) the information necessary to see how serious these concerns are does not seem to be available. (In most cases our assessment is based on significant back-and-forth with the charities themselves, though in some cases we are going off their website.)

• Kiva: commonly perceived as a way to (a) make interest-free loans to (b) entrepreneurs you personally select. In fact, it is neither. (See GiveWell Board member Tim Ogden’s summary of the recent discussion/controversy.)

  We ultimately see a Kiva “loan” as a gift to a microfinance institution, and we have major concerns about supporting any but the most outstanding microfinance institutions as well as concerns about Kiva’s due diligence process specifically.

  Full set of blog posts on Kiva

• Grameen Foundation and other large U.S. microfinance charities: in addition to our general concerns about microfinance charities (and the fact that these charities have not provided the information that would address them), we find that these organizations are rarely even clear about what their value-added is. Also see the disappointing (and eye-opening) evaluation of the Village Phone program.

  Full set of blog posts on Grameen Foundation

• Heifer International: commonly perceived as a way to “give a cow to a poor family as a gift,” but this is in fact a donor illusion -donations support Heifer International’s general “agricultural assistance” activities. We have concerns about this general area and concerns about giving livestock specifically. Neither Heifer’s website nor its grant application have provided the kind of information needed to address these concerns.

  Full set of blog posts on Heifer International

• Smile Train: Prof. Steven Levitt of Freakonomics sees it as using an innovative model to deliver cleft surgeries for $250 apiece. Our analysis suggests that it has “more money than doctors” and that it is therefore out of room for more funding. Smile Train’s recent removal of the charts we questioned and its own explanation of why it is giving grants to other major charities appear to support our analysis. In addition, we have major concerns about surgery charities that Smile Train has not provided enough information to address.

  Full set of blog posts on Smile Train

• UNICEF offers donors the chance to “buy” vaccines and other items. We suspect that this offer is a donor illusion, but the larger issue is that UNICEF provides almost no information at all about its huge variety of activities and how they are going. Most giant international aid charities (Save the Children, World Vision, etc.) are similarly complex and opaque, with CARE coming the closest to meaningful transparency.

  Full set of blog posts on UNICEF

• The Acumen Fund stresses its metrics-driven approach to “patient capital,” investing in businesses that aim for long-term sustainability (i.e., profitability) and social impact. As of now, we see no substantive available information about its track record.

  Full discussion

• The Robin Hood Foundation also stresses its emphasis on metrics, quantification, and analytical thinking. But we have found that it is unwilling to share any information about the results of its grantmaking. (Note that it also pushes the “0% overhead” donor illusion.)

  Full set of blog posts on The Robin Hood Foundation

• The Millennium Villages project states that it is “demonstrating that it’s possible to escape … extreme poverty” (front page of its website). As far as we can tell, very few results have been demonstrated to date, with a few boosts in crop yields as its main evidence of impact - evidence we feel is highly questionable.

  Full discussion

• The Worldwide Fistula Fund has been praised by Nicholas Kristof as “giving new life to the pariahs.” Our assessment is that the Fund is pushing a debatable agenda whose effectiveness has yet to be shown.

  Full discussion

• The Carter Center is, in our view, by far the “best bet” (for an individual donor, our focus) of the charities listed in this post. It has achieved substantial, demonstrable impact in the past through its work on guinea worm eradication, and unlike all of the charities listed above, it publicly discloses substantive technical information on the impact of its past activities.

  However, there is a crucial piece of information missing: how it allocates its budget between programs (in particular, how much it spends on its stronger and better-documented vs. more questionable programs). It has declined to share this (basic, in our view) information in repeated exchanges.

  Full set of blog posts on The Carter Center

We think the above charities are fairly representative of “average” charities in international aid. Some tell better stories than others and some have more disclosure than others. But in almost all cases, international aid charities are (a) carrying out complex projects that can fail to do good (or even do harm) in a variety of ways, and (b) not systematically sharing the information that would make it possible to assess how their work is going.

GiveWell is devoted to finding charities in which we can have more confidence. We’ll be discussing our two top-rated charities working internationally in forthcoming posts.

We recently argued that Smile Train has “more dollars than doctors” for its core program. In that light, yesterday’s Virginian-Pilot article (which quotes me) is interesting:

• The main story is that Smile Train has been trying to make substantial and unrestricted grants to another major cleft surgery charity, Operation Smile. This despite the fact that Smile Train has in the past raised concerns about Operation Smile (”intentionally fabricated tens of thousand of surgeries… distorted its financial reporting… squandered millions of dollars… provided shoddy medical care….”)
• The story quotes Smile Train’s President as appearing to explicitly support the “more dollars than doctors” idea: “Smile Train focused on funding operations by doctors in the countries in need … Mullaney concedes, though, that in some countries, such as Somalia and Haiti, the need outstrips the number of surgeons available to do the work.”

It’s hard to make sense of Smile Train’s wish to make unrestricted grants to Operation Smile, except by accepting that Smile Train is out of room for more funding in its core program.

Possibly, Smile Train’s concerns about Operation Smile have been addressed. Arguably, the decision to grant extra funds to other organizations is admirable (we don’t know whether other charities respond to the same situation by simply piling up assets). But it certainly seems difficult to argue that Smile Train’s donors should think of themselves as funding more of the “$250 per surgery” core program.

If a charity demonstrates that its core program has changed lives in the past, is likely to change lives in the future, and gets great “bang for your buck,” is this enough reason to donate to it? We say no.

The missing piece: Will more funding lead to more of the good program(s)? We generally call this the “room for more funding” question, and we’ve seen next to no helpful discussion of the issue within academia, within the nonprofit sector, or anywhere else.

Often, when I raise this issue, the response I get is “But is that a real problem? Are there charities that have great programs they can’t or won’t expand with more funding?” The answer is yes. Examples:

• Our analysis of Smile Train strongly suggests that its core program of directly funding doctors has “more money than doctors.” Thus, over 50% of Smile Train’s funds go to activities far from what their fundraising focuses on, including grants to other organizations, research, and “provid[ing] materials on cleft lip and palate for free to anyone interested in this birth defect.” Perhaps these activities have value, but it would be a mistake to donate to Smile Train just based on their headline program.
• The Aravind Eye Care System is one of the more impressive humanitarian organizations we have seen, performing vision-restoring surgery extremely cost-effectively. They have been so successful, in fact, that their core program doesn’t need donations - as they have explicitly told us. Revenue from for-pay surgeries subsidizes free surgeries, and donations are used on entirely and substantially different programs such as distribution of spectacles and free food.
• Today’s Aid Watch post gives an excellent picture of why it’s so important to be wary:
  

  according to Fred Martin, Communications Director at CHF, “In fact our Food Pak program is a small portion of what we do. We highlight it because it is our flagship program that we’ve seen work very well in building relationships with the poor so that deeper needs can be uncovered and responded to.” I learned from Fred they also provide beds in eastern Europe and medicines in Asia …

  As long as charities can get away with it, their incentive is to advertise the best program they have, even well beyond the point where that’s the program that needs more money.

There can be many bottlenecks to expanding a program besides money (skilled labor, environments that are conducive to the program, etc.) If you want to fund great programs, you have to ask not just “What have you done and has it worked?” but “What will you do with more funds than you’re currently expecting?”

We haven’t identified any easy answer or simple formula for this question. We believe that “restricting” your donation to the program you favor is generally a futile endeavor (more on this in a future post).

We have developed some relevant ideas. In addition to some rules of thumb for avoiding the most tangled cases, we ask the strongest charities for documents that speak to the “room for more funding” issue directly, such as examples of un-funded but strong project proposals and financial “scenario analysis” (details to come in future posts).

However, we have found that requesting such documents is an uphill battle because the request is generally so foreign. You won’t see financial scenario analysis on any standard list of “documents a charity should be sharing” (from the IRS or anyone else).

Foundations arguably don’t need to deal with the challenge discussed here, because they can give money in large enough chunks to dictate which projects get carried out. (The extent to which this practice is wise is another question). This may be why no one else seems to be asking for information on “room for more funding.” Whatever the reason, it’s an issue that needs much more attention than it’s getting.

For an organization focused on financial metrics, the American Institute of Philanthropy can be very interesting. I can’t do justice to this excellent article on Smile Train with an excerpt, and I urge you to read it all.

It thoroughly debunks an alleged claim by Smile Train that “100% of your donation goes toward programs — 0% goes toward overhead.” Smile Train currently seems to have backed off this claim at least somewhat, although Steven Levitt of Freakonomics appears to have been sold this story (and to have bought it) in 2006.

If identifying effectiveness with “low overhead” is silly, the idea of “0% overhead” simply seems absurd. As the article shows, it doesn’t (and can’t) mean that there are no operating costs affecting the total costs of the program. Rather, it’s another case of zooming in on “your” money, rather than discussing the true total costs of the program you’re supporting the existence of. It makes no sense in an analytical framework; it’s a feel-good gimmick.

That’s why we were surprised when we first saw this gimmick prominently displayed by a group that many consider to be the epitome of hard-nosed, analytical giving: Robin Hood.

Robin Hood’s financials make the situation look similar to Smile Train’s (minus the questionable reclassification of funds that AIP attributes to the latter). About 11% of Robin Hood’s total expenses are “Administration salaries and overhead” or “Fundraising and Public Information,” but because Board member donations are earmarked to those expenses, everyone else can be told their donations are “overhead-free.”

If your goal were to minimize overhead, the fact that Robin Hood tags funds this way shouldn’t be very relevant to you. Robin Hood could allocate more of those Board donations to programs if it spent less on overhead. If you gave to another organization, you could be scaling up an overall lower-overhead operation.

Bottom line: The “0% overhead” claim is promoting the wrong metric (low overhead) and offering a false way to accomplish it.

Two days ago, I critiqued the message sent by charts on Smile Train’s “Financial Information” page, particularly the idea that a surgery was being provided for every $250 in donations.

My recollection is that the page had been up in this form since at least the summer of 2006 (when I first started investigating surgery charities), and Internet Archive appears to confirm that it’s been up at least since early 2007.

We have not heard/seen a response from Smile Train, but as of last night, that page appears to have been taken down entirely.

Smile Train is still claiming $250 as the cost per surgery, which (I argued) seems contradicted by the now-taken-down charts.

Details

Two pages appear to have changed: the “charts” page (the page I critiqued) is now gone, and the “financials” page now longer links to the “charts” page.

“Charts” page:

• 11/25/09 Google cache (this link won’t be good long-term) and screenshots of Google cache: top half, bottom half
• The URL now goes to a “not found” page
• Full page as it originally appeared

“Financials” page:

• 11/27/09 Google cache (won’t be available long-term) and screenshot of Google Cache. Includes a “Click to see our latest financial information” link.
• The current version of the page does not include a “Click to see our latest financial information” link.

These charts from Smile Train imply an appealing story:

(a) Smile Train performs surgeries for $250 apiece.
(b) Smile Train’s main use of donations is to fund $250 surgeries.
(c) A donation to Smile Train funds more surgeries than would the same donation to another cleft palate organization.
(d) If Smile Train had much more money, it would fund many more $250 surgeries.

But after some basic investigation of their website and financials (and conversation with the organization), we believe that:

(a) Smile Train’s total expense per surgery is well over $250.
(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train’s expenses.
(c) Smile Train makes direct grants to other cleft palate organizations, some of which use the same model as the “mission groups” that its charts imply are overly costly.
(d) If Smile Train had more money, it could mean more grants to “mission groups,” general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.

We feel that, as a result, many of Smile Train’s donors - including Prof. Steven Levitt of Freakonomics and Superfreakonomics - have a highly inaccurate picture of how their donations are used. Details follow.

(a) Smile Train’s total expense per surgery is well over $250.

The same Smile Train page linked above claims 90,000-95,000 surgeries provided in 2008. If this number is entirely correct (not exaggerated or estimated) and we divide it into the roughly $100 million of “money raised” shown above, we obtain a cost-per surgery of over $1000.

In fact, the situation is a bit more complex, because money raised doesn’t equal money spent. To get the latter, we can look at Smile Train’s most recent IRS Form 990, which puts total expenses around $85 million for an implied cost-per-surgery closer to $900.

It’s possible that some surgeries are performed for $250 apiece or less. But are these the surgeries you’re paying for when you donate? That’s where the next few points become relevant.

(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train’s expenses.

Here’s the big picture on Smile Train’s expenses, as far as we can understand them from the most recent IRS Form 990:

“Treatment partnerships” appears to be the core program of funding developing-world doctors to perform surgeries.

The “No information available on website” item refers to $23.6 million spent on “provid[ing] materials on cleft lip and palate for free to anyone interested in this birth defect” (according to the Form 990). We have no other information about this activity.

“Education grants” appear to refer to training developing-world surgeons. “Treatment grants” appear to correspond to grants to other organizations.

(c) Smile Train makes direct grants to other cleft palate organizations, some using the same model as the “mission groups” that its charts imply are overly costly.

Prof. Steven Levitt of Freakonomics and Superfreakonomics writes,

The typical model for cleft repair in developing countries has been to convince U.S. doctors and nurses to volunteer a week’s time, fly to a country, and do 80 surgeries … Smile Train instead partners with and trains local doctors to do the surgeries, which turns out to be far more efficient.

In fact, Smile Train makes grants to a variety of other organizations, including some that focus on the “mission” model (flying U.S. doctors and nurses overseas). The most recent list of grants starts on page 18 of the Form 990, and includes:

• $1.4 million to GiveWell standout Interplast, which funds both missions and work with local doctors.
• $70,000 to Surgical Volunteers International and $25,000 to the Smiles International Foundation, both of which appear to be missions only.
• A wide variety of other grants, including for organizations focused on research, organizations that work primarily in the U.S., and more.

(d) If Smile Train had more money, it could mean more grants to “mission groups,” general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.

I first encountered some of these oddities back in 2006, and wondered why Smile Train wasn’t simply putting all available funds toward its most cost-effective surgeries (presumably, the ones that cost $250 each). The answer I got at that time was that there was a “doctor shortage” - Smile Train was paying $250 per surgery as much as it could, but had more money than it could use that way, so it engaged in other activities and made grants to other organizations.

Given this situation, I think Smile Train’s approach makes sense. How many charities are raising more cash than they can productively use, and sitting on it instead of giving it away to other organizations?

In terms of spending money it has appropriately, Smile Train may be doing well. But how it brings in that money is another question. When Steve Levitt doesn’t seem aware of how funds are being used, it seems to me that donors are getting the wrong impression.

Prof. Levitt notes that “One thing they don’t do, but maybe they should do, is literally link each $250 donation to a particular child and send before and after pictures.” Smile Train can’t take this suggestion, because it’s not providing a surgery for every $250 raised.