The GiveWell Blog

What we know about Robin Hood (almost nothing)

(updated on: ) | by Holden

One of the charities we’re often asked about is the Robin Hood foundation. Partly because we used to work at a hedge fund and Robin Hood is big in the hedge fund world; partly because we emphasize analytical, impact-focused giving and Robin Hood has a reputation for analytical, impact-focused giving.

Robin Hood works mostly in the area of U.S. equality of opportunity. We believe this is a very difficult area where well-intentioned and well-executed programs often fail.

Robin Hood’s website does not appear to link to any evidence regarding its impact. Its content seems like that of a typical charity to us, heavy on anecdotes and making use of the “0% overhead” gimmick.

We have asked Robin Hood many times for evidence regarding its impact. We have not only called Robin Hood but have, on more than one occasion, gotten to know Robin Hood donors (giving tens of thousands of dollars) who have asked on our behalf for evidence of impact. We have not been provided with any evidence regarding impact.

We have been provided, on more than one occasion, with a paper on Robin Hood’s conceptual approach to metrics, but this paper discusses how charities would be evaluated given a set of hypotheticals and does not discuss any actual evidence of impact from Robin Hood grantees.

At one time, I was able to have a phone conversation with a high-up staff member at Robin Hood. (I believe it was Michael Weinstein, but I’m not sure – this was before GiveWell was a full-time project, and I didn’t keep good records.) What I am allowed to say about the conversation:

  • I asked if Robin Hood could share any evidence regarding its impact. I was told that all information regarding grantee performance is confidential.
  • I asked if Robin Hood might share its general views about the most promising program types, without discussing specific grantees. I was told that Robin Hood does not intend to publish such views because they would be held to high standards of academic rigor, standards he felt were not appropriate for the practical work that Robin Hood does. (As a side note, I believe organizations like Poverty Action Lab and Innovations for Poverty Action to be both academically rigorous and practically effective.)
  • He did informally share some views on effective programs with me, but asked that I keep the specifics of these views confidential.
  • I asked if he might informally advise us in our research. I was told that time constraints did not permit it.

Despite this extreme commitment to confidentiality, we have ended up seeing some evaluation materials from Robin Hood. The reason is that its grantees have sent them to us as part of their applications for our funds. In general, we have found these materials to provide insufficient evidence to determine impact, much less quantify it in the way Robin Hood’s “metrics” paper describes.

  • Groundwork Inc. submitted two reports from Philliber Research Associates that its main application stated were funded by Robin Hood. (All three documents are available at that link.) These reports report on student improvement on various metrics without addressing “counterfactual”-relevant questions such as “What would be expected of nonparticipant students by these same measures?”
  • SEIU-League Grant Corporation submitted a Robin Hood progress report (available at that link) discussing training program completion and job placement and retention (though data on the latter was not yet available).
  • Other documents were sent for our eyes only, including one that one appeared to directly compare job placement/retention figures across organizations serving seemingly significantly different populations.
  • More generally, we’ve independently evaluated many Robin Hood grantees and found insufficient evidence of impact (see, for example, our report on employment assistance, which includes many Robin Hood grantees).

Bottom line

If I were to guess what I think of Robin Hood’s methodology, I would guess that it is much more focused on quantifying good accomplished – and less focused on being skeptical about whether there was a real impact – than I am. I would defend my view by pointing to past programs in similar areas shown to have no or negligible effects (despite apparent effects using simpler, more selection-bias-prone methodologies). I would argue that rewarding proven impact provides incentives to make sure lives are really being changed for the better, while focusing on quantification provides incentives for charities to embrace selection bias and other subtle factors that can skew non-rigorous studies in their favor. I would also argue that Robin Hood is not helping the populations that could benefit most from its funds.

But I don’t have much to go on. What I am much more confident of is that Robin Hood has essentially no transparency, and essentially no accountability, to the public and to its donors (at least the smaller donors, i.e., those giving tens of thousands of dollars).

Robin Hood, Smile Train and the “0% overhead” donor illusion

(updated on: ) | by Holden

For an organization focused on financial metrics, the American Institute of Philanthropy can be very interesting. I can’t do justice to this excellent article on Smile Train with an excerpt, and I urge you to read it all.

It thoroughly debunks an alleged claim by Smile Train that “100% of your donation goes toward programs — 0% goes toward overhead.” Smile Train currently seems to have backed off this claim at least somewhat, although Steven Levitt of Freakonomics appears to have been sold this story (and to have bought it) in 2006.

If identifying effectiveness with “low overhead” is silly, the idea of “0% overhead” simply seems absurd. As the article shows, it doesn’t (and can’t) mean that there are no operating costs affecting the total costs of the program. Rather, it’s another case of zooming in on “your” money, rather than discussing the true total costs of the program you’re supporting the existence of. It makes no sense in an analytical framework; it’s a feel-good gimmick.

That’s why we were surprised when we first saw this gimmick prominently displayed by a group that many consider to be the epitome of hard-nosed, analytical giving: Robin Hood.

Robin Hood’s financials make the situation look similar to Smile Train’s (minus the questionable reclassification of funds that AIP attributes to the latter). About 11% of Robin Hood’s total expenses are “Administration salaries and overhead” or “Fundraising and Public Information,” but because Board member donations are earmarked to those expenses, everyone else can be told their donations are “overhead-free.”

If your goal were to minimize overhead, the fact that Robin Hood tags funds this way shouldn’t be very relevant to you. Robin Hood could allocate more of those Board donations to programs if it spent less on overhead. If you gave to another organization, you could be scaling up an overall lower-overhead operation.

Bottom line: The “0% overhead” claim is promoting the wrong metric (low overhead) and offering a false way to accomplish it.

Comments on this blog

| by Holden

Lately we have seen a surge in thoughtful and interesting comments on this blog. To those participating in the discussions, thank you and please keep them coming.

We try to respond to any comment that is substantive and critical, though when there are as many as there have been lately, we may go a few days at a time without responding. I’ve just posted responses to the latest batch, excluding the comments on our latest post (“a conflict of Bayesian priors”) which I will get to later.

A conflict of Bayesian priors?

(updated on: ) | by Holden

This question might be at the core of our disagreements with many:

When you have no information one way or the other about a charity’s effectiveness, what should you assume by default?

Our default assumption, or prior, is that a charity – at least in one of the areas we’ve studied most, U.S. equality of opportunity or international aid – is falling far short of what it promises donors, and very likely failing to accomplish much of anything (or even doing harm). This doesn’t mean we think all charities are failing – just that, in the absence of strong evidence of impact, this is the appropriate starting-point assumption.

Many others seem to have the opposite prior: they assume that a charity is doing great things unless it is proven not to be. These people are shocked that we hand out “0 out of 3 stars” for charities just because so little information is available about them; they feel the burden of proof is on us to show that a charity is not accomplishing good. When someone asks them to give to a charity, they usually give.

This puts us in an odd situation: we have very little interest in bad charities, yet others are far more interested in us when we talk about bad charities. To us, credible positive stories are surprising and interesting; to others, credible negative stories are surprising and interesting.

A good example is Smile Train. Nothing in our recent post is really recent at all – we did all of that investigation in 2006. We’ve known about the problem with Smile Train’s pitch for over three years, and have never written it up because we just don’t care that much.

Since day one of our project, our priority has been to identify top charities. When we see a promising charity, we get surprised and excited and pour time into it; when a charity looks bad, we move on and forget about it. Yet others find a report on a great charity to be mind-numbing, and are shocked and intrigued when they hear that a charity might be failing.

So which prior is more reasonable? Before you have evidence about a charity’s impact, should you assume that it’s living up to its (generally extravagant) promises? Or should you assume that it’s well-intentioned but failing?

We’ll offer up a few points in defense of our position in a future post. If you have the opposite position, please share your thoughts in the comments.

When is a charity’s logo a donor illusion?

(updated on: ) | by Holden


When the charity is Nothing But Nets.

Peter Singer has explained the problem with the “net = life” equation, and any other serious analysis we’ve seen of insecticide-treated nets agrees.

Why does this matter? Because Nothing But Nets also prominently states that the total cost of each net is $10. For donors looking to maximize “bang for the buck,” $10 per life saved would probably be the best option available – if it were a real option.

As it happens, distributing bednets is one of the most cost-effective programs we know of (at least when it works) – it’s just nowhere near the $10 figure.

There are probably many reasons that impact-focused giving is so rare, but we think that at least one factor is that when a donor does go looking for information, so many of the “numbers” and “facts” they run into are exaggerations and illusions. It’s a frustrating experience that leaves some jaded about the whole endeavor, and thus missing out on real chances to have an enormous impact.

By default, assume aid projects aren’t reaching the poorest

(updated on: ) | by Holden

If you don’t have evidence one way or the other, should you assume an aid’s projects benefits are reaching the poorest?

We think it’s fair to assume the people with the most need are the people with the least power. We’d also guess that, in general, the people with the most power are best positioned to get anything valuable (training, materials, loans, or whatever else) a charity is subsidizing.

There are ways a charity can get around this dynamic, such as:

  • Working in an area where everyone is in need. We believe that such areas exist, but just because an area has low average incomes, high disease rates, etc. doesn’t mean it has no relatively privileged and powerful members.
  • Running programs that are only appealing to those in need. Some health programs work this way (it’s hard to be treated for tuberculosis unless you have tuberculosis). Microfinance may work this way when interest rates are competitive with (not highly subsidized relative to) alternative sources of credit.
  • Carefully targeting those in need. We should expect this to be very difficult. Charity is inherently about coming into a community from the outside; targeting those in need will generally mean trying to outmaneuver at least some locals. The more locals genuinely share the charity’s mission, the better, but how is the charity to know which it’s dealing with?

Unfortunately (as usual), there isn’t much information out there about how often charities actually succeed in targeting those in need. In conducting our grant application process, we’ve found that systematic assessment of this question is relatively rare. But there is certainly room for concern, as shown by a World Bank review that we’ve quoted before:

The frequent tendency for participatory projects to be dominated if not captured by local elites is highlighted by several case studies. Katz and Sara (1997), in a global review of water projects, find numerous cases of project benefits being appropriated by community leaders and little attempt to include households at any stage … even well trained staff are not always effective in overcoming entrenched norms of exclusion. In a study of community forestry projects in India and Nepal that worked reasonably well, Agarwal (2001) reports that women were systematically excluded from the participatory process because of their weak bargaining power. Rao and Ibanez (2003) find that in the participatory projects in their Jamaican case study, wealthier and better networked individuals dominated decision making. In a similar case-based evaluation of social funds in Jamaica, Malawi, Nicaragua, and Zambia, the World Bank (2002) Operations Evaluation Department concludes that the process was dominated by “prime movers.”

Abraham and Platteau (2004) present evidence on community participation processes in Sub-Saharan Africa based largely on anecdotal evidence from their work in community-based development and on secondary sources. They argue that rural African communities are often dominated by dictatorial leaders who can shape the participation process to benefit themselves because of the poor flow of information. (40-41)