The GiveWell Blog

Smile Train

(updated on: ) | by Holden

These charts from Smile Train imply an appealing story:

(a) Smile Train performs surgeries for $250 apiece.
(b) Smile Train’s main use of donations is to fund $250 surgeries.
(c) A donation to Smile Train funds more surgeries than would the same donation to another cleft palate organization.
(d) If Smile Train had much more money, it would fund many more $250 surgeries.

But after some basic investigation of their website and financials (and conversation with the organization), we believe that:

(a) Smile Train’s total expense per surgery is well over $250.
(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train’s expenses.
(c) Smile Train makes direct grants to other cleft palate organizations, some of which use the same model as the “mission groups” that its charts imply are overly costly.
(d) If Smile Train had more money, it could mean more grants to “mission groups,” general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.

We feel that, as a result, many of Smile Train’s donors – including Prof. Steven Levitt of Freakonomics and Superfreakonomics – have a highly inaccurate picture of how their donations are used. Details follow.

(a) Smile Train’s total expense per surgery is well over $250.

The same Smile Train page linked above claims 90,000-95,000 surgeries provided in 2008. If this number is entirely correct (not exaggerated or estimated) and we divide it into the roughly $100 million of “money raised” shown above, we obtain a cost-per surgery of over $1000.

In fact, the situation is a bit more complex, because money raised doesn’t equal money spent. To get the latter, we can look at Smile Train’s most recent IRS Form 990, which puts total expenses around $85 million for an implied cost-per-surgery closer to $900.

It’s possible that some surgeries are performed for $250 apiece or less. But are these the surgeries you’re paying for when you donate? That’s where the next few points become relevant.

(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train’s expenses.

Here’s the big picture on Smile Train’s expenses, as far as we can understand them from the most recent IRS Form 990:

“Treatment partnerships” appears to be the core program of funding developing-world doctors to perform surgeries.

The “No information available on website” item refers to $23.6 million spent on “provid[ing] materials on cleft lip and palate for free to anyone interested in this birth defect” (according to the Form 990). We have no other information about this activity.

“Education grants” appear to refer to training developing-world surgeons. “Treatment grants” appear to correspond to grants to other organizations.

(c) Smile Train makes direct grants to other cleft palate organizations, some using the same model as the “mission groups” that its charts imply are overly costly.

Prof. Steven Levitt of Freakonomics and Superfreakonomics writes,

The typical model for cleft repair in developing countries has been to convince U.S. doctors and nurses to volunteer a week’s time, fly to a country, and do 80 surgeries … Smile Train instead partners with and trains local doctors to do the surgeries, which turns out to be far more efficient.

In fact, Smile Train makes grants to a variety of other organizations, including some that focus on the “mission” model (flying U.S. doctors and nurses overseas). The most recent list of grants starts on page 18 of the Form 990, and includes:

  • $1.4 million to GiveWell standout Interplast, which funds both missions and work with local doctors.
  • $70,000 to Surgical Volunteers International and $25,000 to the Smiles International Foundation, both of which appear to be missions only.
  • A wide variety of other grants, including for organizations focused on research, organizations that work primarily in the U.S., and more.

(d) If Smile Train had more money, it could mean more grants to “mission groups,” general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.

I first encountered some of these oddities back in 2006, and wondered why Smile Train wasn’t simply putting all available funds toward its most cost-effective surgeries (presumably, the ones that cost $250 each). The answer I got at that time was that there was a “doctor shortage” – Smile Train was paying $250 per surgery as much as it could, but had more money than it could use that way, so it engaged in other activities and made grants to other organizations.

Given this situation, I think Smile Train’s approach makes sense. How many charities are raising more cash than they can productively use, and sitting on it instead of giving it away to other organizations?

In terms of spending money it has appropriately, Smile Train may be doing well. But how it brings in that money is another question. When Steve Levitt doesn’t seem aware of how funds are being used, it seems to me that donors are getting the wrong impression.

Prof. Levitt notes that “One thing they don’t do, but maybe they should do, is literally link each $250 donation to a particular child and send before and after pictures.” Smile Train can’t take this suggestion, because it’s not providing a surgery for every $250 raised.

Poor in the U.S. = rich

(updated on: ) | by Holden

A single-parent family of three in New York, making $8000 per year, makes under half the income level of the Federal poverty line and qualifies for food stamps, TANF (direct cash benefits) and Medicaid. (Details at our guide to U.S. public assistance)

And yet, at $2,667 per person per year, this family is wealthier than 70% of the people in the world. (See the Global Rich List calculator as well as the Giving What We Can version, which may be using more up-to-date data.)

In the poorest parts of the world, fewer than half have access to a latrine or toilet; only 17% own a television; and 19-45% lack access to a reliable source of clean water. In the U.S., practically everyone has all three. (Details.) As we wrote yesterday, the two areas have completely different concepts of “hunger.” And finally, while anyone in the U.S. can ultimately be served in an emergency room, people across the world die or suffer from health conditions for which proven solutions exist.

In the U.S., helping the less fortunate usually means tackling a thorny “equality of opportunity” problem such as improving education or helping struggling adults to find and retain jobs. These problems have a long history of failure and few proven approaches to them. By contrast, helping people overseas can mean delivering something as proven and life-changing as a bednet or tuberculosis treatment.

Hopefully, these observations give some context on why a charitable dollar goes so much further overseas. If you are looking to use your wealth to help those less fortunate, we believe it’s hard to argue the case for U.S. as opposed to international charity, unless you believe that American lives are orders of magnitude more valuable.

Hunger here vs. hunger there

(updated on: ) | by Holden

There has been a fair amount of buzz lately (examples here, here, here, here) about “food insecurity” in the U.S. According to the Reuters headline, one in seven Americans is short of food. In looking into the data, what has surprised us is how different the meaning of “hunger” is when we’re talking about the U.S. vs. the developing world.

Developing-world hunger: 30% of children underweight

Developing-world hunger is usually discussed in terms of incredibly severe indicators of hunger. For example, a 2008 Lancet study estimates that

  • 32% of under-5 children in developing countries are “stunted” (height-for-age severely below normal, such that only 2.3% of children should be “stunted” in a normal distribution)
  • 20% are “underweight” (weight-for-age severely below normal, such that only 2.3% of children should be “underweight” in a normal distribution)

  • 3.5% are “wasted” (weight-for-age even more severely below normal, such that only 0.13% of children should be “wasted” in a normal distribution).

These are indicators of severe, long-term consequences of constant undernutrition for young children.

Food insecurity in the U.S. means anything other than complete and constant food security

The recent USDA release that has formed the basis of the recent discussion of U.S. hunger states:

Eighty-five percent of American households were food secure throughout the entire year in 2008, meaning that they had access at all times to enough food for an active, healthy life for all household members. The remaining households (14.6 percent) were food insecure at least some time during the year, including 5.7 percent with very low food security—meaning that the food intake of one or more household members was reduced and their eating patterns were disrupted at times during the year because the household lacked money and other resources for food. (Emphasis mine)

The report’s summary specifies that food insecurity is usually temporary as opposed to chronic (pg 9) and that children are usually protected from food insecurity even in food-insecure households (pg 6-7).

The “food insecurity” categories are derived from people’s answers to questions like “We worried about whether our food would run out before we got money to buy more” and “We couldn’t afford to eat balanced meals” (full list on pg 3). The details of the answers are found on page 45:


These data imply that anything approaching the sort of hunger measured in the developing world is practically nonexistent in the U.S.

Note in particular the difference regarding children. In the developing world, as shown above, severe child hunger is rampant. In the U.S., even in “food insecure” families, it’s extraordinarily rare for children to go hungry even temporarily. And indeed, World Bank data estimates that 1.3% of U.S. children under 5 are “underweight” – less than the 2.3% that would be expected in a fully normal distribution.

Also note that the USDA report estimates costs for different levels of “food plans” (pg 55), and that its “Thrifty” plan – the cheapest – ranges from $21-$40 per week ($3-$6 per day) depending on age. In the developing world, meanwhile, over 2.5 billion people are estimated to live on less than US$2.50 a day total.

Bottom line

We have no intention of trivializing the situation of those in poverty in the U.S. But for a donor making choices, it can be stunning to see what a different meaning “hunger” takes on when applied at home vs. abroad. Do you value the lives of Americans so much more that you’d rather help people with the second kind of hunger than people with the first?

Acumen Fund and social enterprise investment

(updated on: ) | by Holden

Seth Godin makes an appealing case for “social enterprise investment” along the lines of The Acumen Fund:

When two people trade, both win. No one buys a bar a soap unless the money they’re spending for the soap is worth less to them than the soap itself.

When someone in poverty buys a device that improves productivity, the device pays for itself (if it didn’t, they wouldn’t buy it.) So a drip irrigation system, for example, may pay off by creating two or three harvests a year instead of one.

…

How does Acumen Fund create these markets? The answer is patient capital. The companies that are selling solar lamps to replace kerosene or water purification systems in tiny villages, or housing projects for peasants in Pakistan or even ambulance services in Mumbai fully intend to make a profit, but the venture capitalists on Sand Hill Road aren’t in a hurry to invest in them. The investments are a little too risky, take a little too long or a little too unproven to take a chance on.

So Acumen finds these entrepreneurs on site in the developing world, funds them, teaches them and pushes them to build really big organizations. A to Z has literally thousands of people in their modern factory creating malaria bed nets in Tanzania. And so it grows.

There are reasonable arguments that this sort of larger-scale financing is far more promising than microfinance, which typically focuses on tiny microenterprises that are unlikely to achieve large scale and create jobs. If we could find a strong charity in this area, we’d be excited to recommend it. But as with other areas, we try to go beyond the story and ask critical questions, and so far we have found no organization – including the Acumen Fund – that appears to answer them.

Our main question in this area pertains to an investor’s track record. We’d expect large-scale investing to have a success rate well under 50%, but we’d want to see at least an example or two of real success in the past, consisting of any of the following:

  • An investment that has been shown to improve the lives of customers (for example, tracking bednet use in areas served by A to Z).
  • An investment that has been shown to create jobs, i.e., employ people with previously low incomes.
  • A financial return. (Note that it isn’t enough to point to a “portfolio” member and observe that it’s a successful, self-sustaining business. Anyone can make a loan or grant to an already-successful business; this doesn’t mean they’re having an impact.)

Given Acumen Fund’s reputation for (a) general excellence and (b) focus on evaluation (see the Acumen Fund discussing these issues here and here), we would have expected them to produce such examples. But after evaluating them for our economic empowerment grant, we have concluded that:

  • They do not directly track social impact of the organizations they invest in.
  • They do not have information on the effects of their investments on jobs.
  • They have declined to share information on the performance of their loans, and have not yet exited any equity investments.

A couple of other concerns:

  • At least one of their investments, VisionSpring, has a major charitable solicitation component (for example, they have applied for funding from us). Acumen’s loan could end up getting repaid by donations, which wouldn’t go well with the idea of “patient capital to create self-sustaining institutions.”
  • Their documents are largely at too general and vague a level to give a concrete sense of how their investments are going.

Given this situation, it’s hard to see the organization as more than an experiment at this point. No matter what its team’s credentials, there is the enormous question of how developed-world business experience translates to the goal of making good and socially impactful investments in the developing world.

As with other unproven charities, it seems to us that Acumen should be primarily funded by those who are very close to the relevant people and issues, until it is able to demonstrate not just a good story but some real results. Other similar organizations seem to be similar cases.

The Global Fund and transparency

(updated on: ) | by Holden

We recently complained that “UNICEF provides no information about where the money goes and what projects are in progress.” Some might feel that this complaint comes from unrealistically high standards of transparency, especially for organizations such as UNICEF. How is an organization spending $2.7 billion a year supposed to report its activities?

Our answer would be: “like the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) does.” (Page 55 of its 2008 annual report shows that its budget size is very close to UNICEF’s at $2.7 billion.)

GFATM provides an online program search of all its activities. For any grant it has given (example), you can see (if completed) the grant proposal, grant agreement, and reports on progress. In other words, you can see how much has spent and how (and whether) progress has been tracked.

GFATM recently released the kind of document we have never seen from any other charity approaching its size: an overall evaluation of its activities and impact. Not a general discussion of the organization; not a “meta-evaluation” discussing the quality of past evaluations; a discussion of the overall impact of all of GFATM’s activities across the world. Furthermore, this report was in no way a fundraising document; it was frank about the fact that inadequate evidence exists for GFATM’s impact to date (see the discussion at our review).

GFATM comes under a lot of criticism, even from its own evaluators. We ourselves have many reservations about its work, as our review establishes. But we have seen very few charities – and no other charities approaching its size – that can make as strong a claim to being a transparent organization and a learning organization.

GFATM proves that neither size nor celebrity support need stop a charity from being clear about what it’s working on and how it’s going.

Two charities, one microfinance institution

(updated on: ) | by Holden

We’re looking for a good option for U.S. donors interested in supporting microfinance. We’ve been examining the largest, most prominent U.S.-registered charities in this area: Grameen Foundation, Unitus, Accion, Women’s World Banking, Opportunity International and FINCA. All of these are large organizations that list a variety of “partner” microfinance institutions.

One thing that might surprise a donor in this area is that sometimes the “partnerships” overlap. For example, Women’s World Banking lists Enda Inter-Arabe in its network, and so does Grameen-Jameel (which itself is a joint venture of the Grameen Foundation). Neither organization’s profile mentions the other.

Grameen Koota lists the Grameen Foundation and Unitus (among others) as donors and lists Unitus and Accion as consultants. Unitus’s page on Grameen Koota doesn’t mention Accion; Accion’s doesn’t mention Unitus; the Grameen Foundation sounds like its relationship is currently more consultant-like than donor-like.

As you can see from the profiles linked above – and as we’ve found from talking to the large microfinance charities – most (not all) of their activities consist of consulting or “technical assistance,” rather than founding, directly supporting or investing in microfinance institutions. The relationship between them and their “partners” often appears relatively limited, and it’s difficult to pin down who’s responsible for what.

Focusing on technical assistance and consulting isn’t necessarily a bad thing – perhaps U.S. expertise is just what these institutions need to serve more clients and serve them better. But we aren’t sure this is the case – it’s also possible that the U.S. charities have sprung up because of strong donor demand for the “story” of microfinance, and are consulting because they have more than they can productively spend on investing and creating.

What we know is that it’s inherently hard to gauge a charity’s impact when its main activities consist of free (or heavily subsidized) technical assistance. Even if we had all the information we want about the partner MFIs themselves, we’d still need to understand the extent to which help with business planning or product design made a difference to them.

So far, we haven’t seen many signs that the U.S. charities are critically asking this question, or making it possible to hold them accountable for the answer.