The GiveWell Blog

Why are we always criticizing charities?

(updated on: ) | by Elie

Recently, we’ve criticized (in one way or another) many well-known, presumably well-intentioned charities (Smile Train, Acumen Fund, UNICEF, Kiva), which might lead some to ask: should GiveWell focus on the bad (which may discourage donors from giving) as opposed to the good (which would encourage them to give more)? Why so much negativity and not more optimism?

The fact is, we are very optimistic about what a donor can accomplish with their charity. Donors can have huge impact — save a life, improve equality of opportunity, or improve education. Our research process, and our main website, are (and always have been) built around identifying outstanding charities.

GiveWell hasn’t set a bar that no charity can meet. Six international charities have met and passed the bar. Where most charities fall short, they succeed.

The problem is: because the nonprofit sector is saturated with unsubstantiated claims of impact and cost-effectiveness, it’s easy to ignore me when I tell you (for example), “Give $1,000 to the Stop Tuberculosis Partnership, and you’ll likely save someone’s life (perhaps 2 or 3 lives).” It’s easy to respond, “You’re just a cheerleader” or “Why give there when Charity X makes an [illusory] promise of even better impact?”

We don’t report on Smile Train, Kiva, Acumen Fund, UNICEF, or any others for the sake of the criticism; we write about them to show you how much more you can accomplish with your gift if you’re willing to reconsider where you’re giving this year.

Unless you have strong reason to believe otherwise, I’d recommend you choose a great charity as opposed to one that’s merely better-than average. If you only have a fixed amount to give, why not support the very best?

Smile Train removes charts from website; still claims $250 per surgery

(updated on: ) | by Holden

Two days ago, I critiqued the message sent by charts on Smile Train’s “Financial Information” page, particularly the idea that a surgery was being provided for every $250 in donations.

My recollection is that the page had been up in this form since at least the summer of 2006 (when I first started investigating surgery charities), and Internet Archive appears to confirm that it’s been up at least since early 2007.

We have not heard/seen a response from Smile Train, but as of last night, that page appears to have been taken down entirely.

Smile Train is still claiming $250 as the cost per surgery, which (I argued) seems contradicted by the now-taken-down charts.

Details

Two pages appear to have changed: the “charts” page (the page I critiqued) is now gone, and the “financials” page now longer links to the “charts” page.

“Charts” page:

“Financials” page:

Kudos to Ken Berger

(updated on: ) | by Holden

Charity Navigator’s CEO writes:

We MUST get past the notion of doing the “good work” with no accountability. We MUST get past the idea that nonprofits are too complex or unique to be measured. I have seen it close up for years and it is not a pretty picture. The nonprofit sector must get its act together and make sure it is really helping provide meaningful change in communities and peoples lives. It is life or death for many of those we serve whether we are effective or not. So let’s work together to measure, manage and deliver what is really important to make our world a better place. (Emphasis in original)

I respectfully disagree with those who call this “just talk.” Rather, I’m with Sean, who says: “Ken has put his chips down big and made a huge bet that he’s going to pull this all off.”

For years, Charity Navigator has dominated news stories about how to give effectively, even while the limitations of its methodology have been widely recognized within the nonprofit sector. Now, Ken is not just talking about but committing to doing more. His statement is so public and so clear that there’s no credible way to turn back, for him or his organization. If Charity Navigator doesn’t want to have its own quotes used against it for the rest of its existence, it will have to see its commitment to real, meaningful evaluation through.

We don’t yet know all the details of how Charity Navigator is going to go about this, and if we have issues with the methodology it ends up adopting, you’ll hear about them. But Ken absolutely deserves a “standing ovation” now. He is the issuer of our press release who had the most to lose, and he’s chosen to fight for what he knows is right rather than stand pat.

The worst way to pick a charity

(updated on: ) | by Holden

Today, the most common way that donors evaluate charities – when they evaluate them at all – is by asking questions about financials, such as “How much of my donation goes to programs vs. salaries?” This approach makes no sense.

We’ve argued this point before at length. Picking charities based on the “overhead ratio” is like picking your doctor by the percentage of revenue spent on medicine (more absurdity-highlighting analogies here). The actual reported “overhead ratio” is vaguely defined and generally up to the charities reporting it; the concept of “minimal money on overhead” discourages a lot of good and necessary spending, and is ultimately irrelevant to the question of whether a charity is changing lives.

What today’s press release (PDF) makes clear is just how much agreement there is on these points. Today’s two most prominent sources of financial metrics – GuideStar and Charity Navigator – agree that there needs to be more to the picture, and are working toward more meaningful metrics.

It’s scary to think about how commonly this metric is touted, when it’s recognized as a red herring. It seems to us that people have responded to the extreme scarcity of information on charities by using whatever information is available, whether or not it’s meaningful. Fortunately, alternative ways of evaluating charities are emerging.

GiveWell, Great Nonprofits and Philanthropedia are all new, small organizations. None of us have perfected a methodology yet – no one has – and all of us would benefit from more input. If more donors took a look at these resources, we would finally see the conversation around “Where should I give?” move beyond the distracting and unhelpful question of “Who has the lowest ‘overhead?’”

That would be a huge step toward a world in which donors reward charities for effectiveness in improving lives.

When donations and profits meet, beware

(updated on: ) | by Holden

David Roodman raises the concern that Kiva capital could be effectively “padding profits” at a profitable microfinance institution. He concludes,

If social investors provide capital at prices below commercial rates to enterprises with “double bottom lines” (profit and social benefit), how do the investors assure that their cheap capital isn’t being used to boost just one of those bottom lines?

We feel that this is a major concern, and one that also applies to larger-scale social enterprise investment (see last week’s discussion of Acumen Fund, particularly the part about VisionSpring).

With a for-profit, everyone is looking to get their investment back. With a nonprofit, everyone is (ideally) looking to achieve social impact. But an organization that has both charitable and for-profit investors can get caught in the middle: taking donations, but measuring its success in terms of profits instead of lives changed.

The situation is particularly dangerous when profits are viewed as a “proxy” for social impact, and thus become the only measure looked at. Imagine an enterprise that could sell food below market prices, thanks to support from donations. It could be distorting the local economy (by outcompeting local farmers), failing to reach those truly in need, and ultimately failing to accomplish anything besides selling food for less than it costs to people who turn it around for a profit. And yet, all of this could be consistent with a good bottom line, which would make both the investors and the donors happy.

One way to avoid this problem is to refuse to use profits as a proxy for social impact – to insist on rigorous assessment of whether an enterprise is changing lives, rather than settling for the logic of “if it’s selling it must be helping.” But such rigorous assessment costs money, and exacerbates the already great challenge of turning a profit. Jim Fruchterman’s recent comment illustrates the low likelihood that you’ll see much of this approach working in practice.

Another possible approach is to get very specific about how donations are and aren’t being used. Any sort of “hybrid” organization ought to be able to show a history of using donations to absorb risk, but ultimately creating ventures whose profitability and sustainability does not depend in any way on continued subsidies. Our basic feeling is that demonstrating such a thing would be harder than it sounds, and that we have not yet seen an organization that seems capable of doing so.

Bottom line: it’s difficult to hold an organization accountable unless all its investors are on the same page about what it’s accountable for. Blended value makes perfect sense in theory, but in practice, it seems like a huge challenge that nobody is clearly up to. In the meantime, it may make more sense for businesses to be businesses and charities to be charities.

Smile Train

(updated on: ) | by Holden

These charts from Smile Train imply an appealing story:

(a) Smile Train performs surgeries for $250 apiece.
(b) Smile Train’s main use of donations is to fund $250 surgeries.
(c) A donation to Smile Train funds more surgeries than would the same donation to another cleft palate organization.
(d) If Smile Train had much more money, it would fund many more $250 surgeries.

But after some basic investigation of their website and financials (and conversation with the organization), we believe that:

(a) Smile Train’s total expense per surgery is well over $250.
(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train’s expenses.
(c) Smile Train makes direct grants to other cleft palate organizations, some of which use the same model as the “mission groups” that its charts imply are overly costly.
(d) If Smile Train had more money, it could mean more grants to “mission groups,” general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.

We feel that, as a result, many of Smile Train’s donors – including Prof. Steven Levitt of Freakonomics and Superfreakonomics – have a highly inaccurate picture of how their donations are used. Details follow.

(a) Smile Train’s total expense per surgery is well over $250.

The same Smile Train page linked above claims 90,000-95,000 surgeries provided in 2008. If this number is entirely correct (not exaggerated or estimated) and we divide it into the roughly $100 million of “money raised” shown above, we obtain a cost-per surgery of over $1000.

In fact, the situation is a bit more complex, because money raised doesn’t equal money spent. To get the latter, we can look at Smile Train’s most recent IRS Form 990, which puts total expenses around $85 million for an implied cost-per-surgery closer to $900.

It’s possible that some surgeries are performed for $250 apiece or less. But are these the surgeries you’re paying for when you donate? That’s where the next few points become relevant.

(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train’s expenses.

Here’s the big picture on Smile Train’s expenses, as far as we can understand them from the most recent IRS Form 990:

“Treatment partnerships” appears to be the core program of funding developing-world doctors to perform surgeries.

The “No information available on website” item refers to $23.6 million spent on “provid[ing] materials on cleft lip and palate for free to anyone interested in this birth defect” (according to the Form 990). We have no other information about this activity.

“Education grants” appear to refer to training developing-world surgeons. “Treatment grants” appear to correspond to grants to other organizations.

(c) Smile Train makes direct grants to other cleft palate organizations, some using the same model as the “mission groups” that its charts imply are overly costly.

Prof. Steven Levitt of Freakonomics and Superfreakonomics writes,

The typical model for cleft repair in developing countries has been to convince U.S. doctors and nurses to volunteer a week’s time, fly to a country, and do 80 surgeries … Smile Train instead partners with and trains local doctors to do the surgeries, which turns out to be far more efficient.

In fact, Smile Train makes grants to a variety of other organizations, including some that focus on the “mission” model (flying U.S. doctors and nurses overseas). The most recent list of grants starts on page 18 of the Form 990, and includes:

  • $1.4 million to GiveWell standout Interplast, which funds both missions and work with local doctors.
  • $70,000 to Surgical Volunteers International and $25,000 to the Smiles International Foundation, both of which appear to be missions only.
  • A wide variety of other grants, including for organizations focused on research, organizations that work primarily in the U.S., and more.

(d) If Smile Train had more money, it could mean more grants to “mission groups,” general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.

I first encountered some of these oddities back in 2006, and wondered why Smile Train wasn’t simply putting all available funds toward its most cost-effective surgeries (presumably, the ones that cost $250 each). The answer I got at that time was that there was a “doctor shortage” – Smile Train was paying $250 per surgery as much as it could, but had more money than it could use that way, so it engaged in other activities and made grants to other organizations.

Given this situation, I think Smile Train’s approach makes sense. How many charities are raising more cash than they can productively use, and sitting on it instead of giving it away to other organizations?

In terms of spending money it has appropriately, Smile Train may be doing well. But how it brings in that money is another question. When Steve Levitt doesn’t seem aware of how funds are being used, it seems to me that donors are getting the wrong impression.

Prof. Levitt notes that “One thing they don’t do, but maybe they should do, is literally link each $250 donation to a particular child and send before and after pictures.” Smile Train can’t take this suggestion, because it’s not providing a surgery for every $250 raised.