We’ve just published reviews of all of the organizations that applied for our economic empowerment grant. (Note that we previously announced the grant winners here.)

Whenever we publish new charity reviews, we get some push back from charities who aren’t happy that they’ve received a 0-star review.

Before we published our economic empowerment reviews, we sent non-public drafts to the charities so that they could review them for any inaccuracies. In the case of this grant, the push back was both louder and more consistent than we had heard before, and it led us to evaluate our process: what mistakes we had made, what we could do to rectify the situation, and what we should change to handle a process like this better in the future.

The criticism

The main criticisms we received from reviewed organizations were the following:

1. Your review implies that you’ve conducted an in-depth investigation of our entire organization and concluded that we’re doing a bad job. All you’ve done is determine that our application wasn’t strong enough to win this grant.
2. We didn’t know that information about our organization would be published on your website. We thought we were participating in a grant process, independent of the general evaluation work GiveWell does.

In some cases these criticisms were valid and spurred us to make changes.

What we’ve changed

Regarding criticism #1, we are generally very careful not to say that we have deemed a charity ineffective. Rather, we say that donors who choose to give to a charity should ask the charity to bear the burden of proof that their activities are accomplishing good. Our approach is to list questions we have about a charity’s activities and say that we have been unable to answer them.

Similarly, we’ve never implied in our reviews that we’ve conducted in-depth investigation of every charity we we rate 0-stars. In explaining why we rely on heuristics in our reviews, we write, “Our research is constrained by practical considerations. Our goal is not to be “perfect” in our assessments but rather to provide better information than donors can find anywhere else.”

Nevertheless, we understand that donors coming to one of our 0-star review pages could misunderstand them and think that we’ve deemed a charity’s programs ineffective. To remedy this, we’ve (a) added explanatory notes at the top of each review to further explain what a 0-star ratings means and (b) tried to specify as clearly as possible the steps we’ve taken for each charity’s review.

Regarding criticism #2, in most cases we feel we were extremely clear about our plans to publish materials.

Most charities we emailed were directed to our main grants page and our FAQ for the grant, both of which prominently discuss our process of publishing our full reasoning and analysis. In addition, many of the applicants (including the majority of those who objected after-the-fact to publication) submitted their applications online, clicking a box on our online application which stated, “I agree that with the exceptions specified in the text box above, all submitted materials are considered public information and will be posted on the GiveWell website.”

That said, there were a handful of organizations that we contacted in non-standard ways, and that can make a reasonable case that we didn’t adequately inform them about public disclosure (at least we can’t establish whether we did). We offered these charities the option not only to have their materials omitted, but to have us not disclose the fact that they applied for this grant from our site.

Note that we never publish an organization’s materials without explicit permission nor do we comment on confidential materials without that permission.

What we’ll do in the future

Two key items we recognized in this process are:

• It’s important to be completely clear and explicit with organizations about our plans to publish reviews and materials.
• Our process selects only the very few organizations that can meet our criteria. This means that a 0-star rating is the most likely outcome for an organization we consider. This may provide a disincentive to lesser-known organizations to apply for our grants and invite us to review them (by contrast, more prominent organizations generally appear on our site whether or not they apply).

One idea we’ve had for future grant applications is to create a page, first, for every organization we invite to apply for a grant. This way, the grant process allows organizations to improve their rating; it won’t cause them to receive a rating. It will also make our plans to publish a review crystal clear.

First it was shoes, then shirts. I don’t have much to add on questions like “Are supplies in fact clogging the roads?” and “Are there in fact people in Africa who don’t wear shirts because they can’t find/afford them?” But to me, the argument against in-kind donations is both simple and general (i.e., it doesn’t depend much on the specifics of the goods):

1. It seems nearly always preferable to sell your unwanted belongings and give the proceeds to charity.

I’m not sure how many (if any) people are “desperate for shirts.” But it does seem that an easy test of whether your old shirt is worth anything to anyone is to ask, “Will someone pay for it?”

This makes it likely that your shirt itself will not get to the poorest people – it will get to whoever pays the most for your shirt. But as long as you give to the proceeds to the best charity you can find, this seems like a good outcome. Imagine that a less-poor person is willing to pay $3 for your shirt; a poorer person wants your shirt but can’t (or won’t) pay $3 for it. That implies that the latter person is better off with the actual $3 (or $3 worth of the best charitable services you can find) than with the shirt.

Of course, if you sell, someone has to pay enough not only for the shirt, but for the costs of shipping and logistics. But that’s part of the point. If there’s no one in the world who will pay enough for your shirt to cover shipping costs, that’s a red flag that you’re looking at a wasteful transaction. Throw out the shirt and give cash. (Again: if someone in Africa is not willing or able to pay the shipping costs to get your shirt, that implies that giving them cash or charitable services equivalent to those shipping costs would be more valued than the shirt.)

The 1 Million Shirts campaign has asked for alternative ways to help people. I think “collect the million shirts, sell them off and give the proceeds to an outstanding charity” is a reasonable proposal.

This is also my understanding of how Goodwill works.

It seems that this argument holds for nearly all in-kind donations. There will be exceptions,* especially where a gift is driven by an acute local need for a particular good. But I would guess that such situations are the exception, rather than the rule, because I don’t think most in-kind donations are driven by the needs of the recipients. That brings me to the next point.

2. Just because your gifts are accepted doesn’t mean they’re wanted.

Good Intentions are Not Enough points to a USAID statement on gifts in kind (PDF). I think it’s an eye-opener, particularly this part:

The American public often … attempts to collect and donate commodities, also referred to as “gifts-in-kind” (GIK). GIK are most often inappropriate for relief programs and harmful to the environment and the local culture. They are expensive to transport, relative to the cost of procuring the same commodities locally. GIK use up scarce resources such as transportation routes, warehouse space, and staff time. They can adversely affect the regional economy by competing with similar commodities available locally. And GIK can contribute to negative images of the United States and its disaster response activities. Despite these problems, USAID often faces significant pressure to assist with or fund shipments of GIK to disaster settings. (Emphasis mine)

I think there is a possible analogy to volunteering. Charities have an incentive – and perhaps also pressure – to accommodate people asking them to give in-kind donations. (We’ve even seen a little of it ourselves; despite the abstractness of our own activities, people come to us asking where to donate leftover drugs or equipment, and it’s never fun to reply with “tough luck.”)

Giving away your goods feels better than taking my above suggestion, because it doesn’t force you to confront the difference between what you paid for the goods and what they’re now worth. (Giving away an old shirt feels better than selling it for $1.) And plenty of organizations are interested in delivering these good feelings. There may also be organizations that simply deliver gifts-in-kind without carefully weighing the shipping costs, logistics issues, etc.

We’ve noted before that much of charity seems set up to serve the donor rather than the recipient. Consider this next time you’re about to give old clothes to people who may not have asked for them (even if it’s through a charity that’s claiming they have).

*For example:

• Companies/producers can have reasons, relating to price discrimination, to donate the the things they produce. These reasons can apply to a drug company donating drugs, but not to an individual giving away old belongings, so I won’t elaborate on this point here.
• Sometimes there is an excellent match between what you have to offer and what someone else needs, to the point where (a) the person in need would be close to the highest bidder to buy it; (b) selling rather than donating would create unnecessary middleman/friction costs. (Analogy: if a friend is willing to pay $250 for an old computer that’s worth $300, I’ll take the friend’s offer instead of going through the hassle of selling. On the other hand, if the friend is only willing to pay $50, I’ll sell the computer.)
• There can be tax advantages to in-kind donations.

Note: I posted most of this post last night as a comment.

There has been some interesting back-and-forth in the comments lately between Laura Deaton and me (see the Should Charity Evaluation be “fair”? thread). Laura seems worried that by issuing ratings and a list of “top” charities, we are implying a level of objectivity and reliability that “overpromises” and can mislead donors. I believe it is a somewhat common concern behind many people’s uncomfortability with the growing interest in charity evaluation.

I think it is helpful to think about how reviews, “top” lists, and evaluations are seen in other domains. For example, movies:

• Movie reviewers share both quantified, judgmental “ratings” and (some of) the reasoning behind their ratings.
• Reviews involve substantial judgment calls.
• Some people read the reviews to get a sense of the reviewers’ values, biases, etc. and decide for themselves how much to weigh them. Some people just look at the ratings.
• Movie reviewers will sometimes put out a “X best movies of the year” list, even though they haven’t even seen all (or even a substantial portion of) the movies made in a year.
• Large information providers, including both newspapers and aggregators such as Metacritic/Rotten Tomatoes, will collect the opinions of many movie reviewers in one place and offer them up as a service to people deciding which movies to watch. They will put their name behind the reviews even though they don’t endorse every word of them.
• The whole system seems to be valuable to a lot of people and to be clearly superior to alternatives such as (a) ratings provided without any reasoning; (b) ratings based on purely objective criteria such as “percent of film budget spent on actors’ salaries”; (c) ratings withheld entirely out of concern for misleading overly impressionable viewers.

Similar systems exist for a lot of consumer decisions.

Any metaphor breaks down and I wouldn’t want this one taken too far. But it seems appropriate for the question at hand, which is whether we are “overpromising” by issuing ratings, evaluations and “top charities” lists. Based on what these terms mean in other domains, I think we are closer to overdelivering.

Added 4/21/2010: I realized after writing this that Tactical Philanthropy made a similar argument in 2007.