Philanthropedia has responded to our take on its microfinance report:

• Philanthropedia’s response to GiveWell’s analysis – Part I
  
• Philanthropedia’s response to GiveWell’s analysis – Part II

We find the response to be encouragingly straightforward. For the most part, it agrees with the concerns we have raised and commits to addressing them. Good Intentions are Not Enough argues that the response raises more questions than it answers, and we agree in substance, though to us the most important part of the response is an honest recognition of shortcomings and an expressed intent to improve.

A couple of responses on points where there is some disagreement:

Re: incentives. Our original post argued that Philanthropedia, if it became highly influential under its current model, would create bad incentives for experts (allowing them to continue to keep their thoughts under wraps) and charities (encouraging them to win over experts in ways unrelated to improving their social impact).

• Philanthropedia states, “Even if only 31 experts agreed to publish their bios and stand behind the results, that is exactly 31 more than before.” We disagree that this is a good thing. The experts have not been linked with recommendations (as far as we can tell) – all we know is that they shared their votes, and we know the aggregate result. The problem here is that no one is individually accountable for their own recommendations and reasoning. If it became influential, this model would allow experts to have the benefits of transparency (i.e., influence over individual donations) without the accountability, which we feel would be a net negative impact on the incentives for experts to be transparent.
• Philanthropedia states that it intends to rigorously guard against “gaming of the system.” But the dynamics we are concerned about are less related to “rule-breaking” or outright conflict of interest (for example, a charity pays an expert for a positive rating) than to distortion of ratings in a softer sense. Becoming “popular in a certain crowd” can be accomplished in a lot of ways that have nothing to do with improving impact. That’s why it’s so important that people who recommend charities put as much as possible of their reasoning out in public, where others can ask, “Are these reasons strong enough to explain this person’s support?”

Should donors use Philanthropedia’s current report?

In its current form, we feel that Philanthropedia amounts to a set of recommendations unlinked to either people or reasoning. That’s essentially asking donors to trust anonymous people, which we feel is a bad idea and not helpful for impact-focused donors. (And if donors need a “Who’s Who in microfinance,” that information is already available.)

For context: we are generally sympathetic to the “It’s not perfect, but it’s better than nothing” argument. We are acutely aware that donors have very limited resources today, and we ourselves have generally erred on the side of publishing/sharing our research as quickly as possible. Our research process and our own knowledge have enormous room for improvement (and this was more true when we first published research in November 2007). But we have not gone as far as publishing recommendations while in the middle of our process – we wanted to make sure we could clearly present our criteria and the charities considered, so that people could check our process and hold us accountable if they wished to do so.

There are good arguments for Philanthropedia’s sharing/promoting its research even in its current state, and it’s ultimately Philanthropedia that makes that call.