Note: I posted most of this post last night as a comment.

There has been some interesting back-and-forth in the comments lately between Laura Deaton and me (see the Should Charity Evaluation be “fair”? thread). Laura seems worried that by issuing ratings and a list of “top” charities, we are implying a level of objectivity and reliability that “overpromises” and can mislead donors. I believe it is a somewhat common concern behind many people’s uncomfortability with the growing interest in charity evaluation.

I think it is helpful to think about how reviews, “top” lists, and evaluations are seen in other domains. For example, movies:

• Movie reviewers share both quantified, judgmental “ratings” and (some of) the reasoning behind their ratings.
• Reviews involve substantial judgment calls.
• Some people read the reviews to get a sense of the reviewers’ values, biases, etc. and decide for themselves how much to weigh them. Some people just look at the ratings.
• Movie reviewers will sometimes put out a “X best movies of the year” list, even though they haven’t even seen all (or even a substantial portion of) the movies made in a year.
• Large information providers, including both newspapers and aggregators such as Metacritic/Rotten Tomatoes, will collect the opinions of many movie reviewers in one place and offer them up as a service to people deciding which movies to watch. They will put their name behind the reviews even though they don’t endorse every word of them.
• The whole system seems to be valuable to a lot of people and to be clearly superior to alternatives such as (a) ratings provided without any reasoning; (b) ratings based on purely objective criteria such as “percent of film budget spent on actors’ salaries”; (c) ratings withheld entirely out of concern for misleading overly impressionable viewers.

Similar systems exist for a lot of consumer decisions.

Any metaphor breaks down and I wouldn’t want this one taken too far. But it seems appropriate for the question at hand, which is whether we are “overpromising” by issuing ratings, evaluations and “top charities” lists. Based on what these terms mean in other domains, I think we are closer to overdelivering.

Added 4/21/2010: I realized after writing this that Tactical Philanthropy made a similar argument in 2007.