We’ve just published reviews of all of the organizations that applied for our economic empowerment grant. (Note that we previously announced the grant winners here.)

Whenever we publish new charity reviews, we get some push back from charities who aren’t happy that they’ve received a 0-star review.

Before we published our economic empowerment reviews, we sent non-public drafts to the charities so that they could review them for any inaccuracies. In the case of this grant, the push back was both louder and more consistent than we had heard before, and it led us to evaluate our process: what mistakes we had made, what we could do to rectify the situation, and what we should change to handle a process like this better in the future.

The criticism

The main criticisms we received from reviewed organizations were the following:

1. Your review implies that you’ve conducted an in-depth investigation of our entire organization and concluded that we’re doing a bad job. All you’ve done is determine that our application wasn’t strong enough to win this grant.
2. We didn’t know that information about our organization would be published on your website. We thought we were participating in a grant process, independent of the general evaluation work GiveWell does.

In some cases these criticisms were valid and spurred us to make changes.

What we’ve changed

Regarding criticism #1, we are generally very careful not to say that we have deemed a charity ineffective. Rather, we say that donors who choose to give to a charity should ask the charity to bear the burden of proof that their activities are accomplishing good. Our approach is to list questions we have about a charity’s activities and say that we have been unable to answer them.

Similarly, we’ve never implied in our reviews that we’ve conducted in-depth investigation of every charity we we rate 0-stars. In explaining why we rely on heuristics in our reviews, we write, “Our research is constrained by practical considerations. Our goal is not to be “perfect” in our assessments but rather to provide better information than donors can find anywhere else.”

Nevertheless, we understand that donors coming to one of our 0-star review pages could misunderstand them and think that we’ve deemed a charity’s programs ineffective. To remedy this, we’ve (a) added explanatory notes at the top of each review to further explain what a 0-star ratings means and (b) tried to specify as clearly as possible the steps we’ve taken for each charity’s review.

Regarding criticism #2, in most cases we feel we were extremely clear about our plans to publish materials.

Most charities we emailed were directed to our main grants page and our FAQ for the grant, both of which prominently discuss our process of publishing our full reasoning and analysis. In addition, many of the applicants (including the majority of those who objected after-the-fact to publication) submitted their applications online, clicking a box on our online application which stated, “I agree that with the exceptions specified in the text box above, all submitted materials are considered public information and will be posted on the GiveWell website.”

That said, there were a handful of organizations that we contacted in non-standard ways, and that can make a reasonable case that we didn’t adequately inform them about public disclosure (at least we can’t establish whether we did). We offered these charities the option not only to have their materials omitted, but to have us not disclose the fact that they applied for this grant from our site.

Note that we never publish an organization’s materials without explicit permission nor do we comment on confidential materials without that permission.

What we’ll do in the future

Two key items we recognized in this process are:

• It’s important to be completely clear and explicit with organizations about our plans to publish reviews and materials.
• Our process selects only the very few organizations that can meet our criteria. This means that a 0-star rating is the most likely outcome for an organization we consider. This may provide a disincentive to lesser-known organizations to apply for our grants and invite us to review them (by contrast, more prominent organizations generally appear on our site whether or not they apply).

One idea we’ve had for future grant applications is to create a page, first, for every organization we invite to apply for a grant. This way, the grant process allows organizations to improve their rating; it won’t cause them to receive a rating. It will also make our plans to publish a review crystal clear.