Philanthropedia has responded to our take on its microfinance report:

• Philanthropedia’s response to GiveWell’s analysis – Part I
  
• Philanthropedia’s response to GiveWell’s analysis – Part II

We find the response to be encouragingly straightforward. For the most part, it agrees with the concerns we have raised and commits to addressing them. Good Intentions are Not Enough argues that the response raises more questions than it answers, and we agree in substance, though to us the most important part of the response is an honest recognition of shortcomings and an expressed intent to improve.

A couple of responses on points where there is some disagreement:

Re: incentives. Our original post argued that Philanthropedia, if it became highly influential under its current model, would create bad incentives for experts (allowing them to continue to keep their thoughts under wraps) and charities (encouraging them to win over experts in ways unrelated to improving their social impact).

• Philanthropedia states, “Even if only 31 experts agreed to publish their bios and stand behind the results, that is exactly 31 more than before.” We disagree that this is a good thing. The experts have not been linked with recommendations (as far as we can tell) – all we know is that they shared their votes, and we know the aggregate result. The problem here is that no one is individually accountable for their own recommendations and reasoning. If it became influential, this model would allow experts to have the benefits of transparency (i.e., influence over individual donations) without the accountability, which we feel would be a net negative impact on the incentives for experts to be transparent.
• Philanthropedia states that it intends to rigorously guard against “gaming of the system.” But the dynamics we are concerned about are less related to “rule-breaking” or outright conflict of interest (for example, a charity pays an expert for a positive rating) than to distortion of ratings in a softer sense. Becoming “popular in a certain crowd” can be accomplished in a lot of ways that have nothing to do with improving impact. That’s why it’s so important that people who recommend charities put as much as possible of their reasoning out in public, where others can ask, “Are these reasons strong enough to explain this person’s support?”

Should donors use Philanthropedia’s current report?

In its current form, we feel that Philanthropedia amounts to a set of recommendations unlinked to either people or reasoning. That’s essentially asking donors to trust anonymous people, which we feel is a bad idea and not helpful for impact-focused donors. (And if donors need a “Who’s Who in microfinance,” that information is already available.)

For context: we are generally sympathetic to the “It’s not perfect, but it’s better than nothing” argument. We are acutely aware that donors have very limited resources today, and we ourselves have generally erred on the side of publishing/sharing our research as quickly as possible. Our research process and our own knowledge have enormous room for improvement (and this was more true when we first published research in November 2007). But we have not gone as far as publishing recommendations while in the middle of our process – we wanted to make sure we could clearly present our criteria and the charities considered, so that people could check our process and hold us accountable if they wished to do so.

There are good arguments for Philanthropedia’s sharing/promoting its research even in its current state, and it’s ultimately Philanthropedia that makes that call.

Now that giving season has ended, we will be shifting our priorities and slowing down the pace of blog posts. Here’s a quick overview of the highlights from our last few months:

• You Can Save A Life: GiveWell in a nutshell. Why give to a charity with a great story and nebulous impact, when by giving to a top charity you can actually save a life?
• Frank discussions of our top two charities, VillageReach and Stop Tuberculosis Partnership.
• Celebrated charities we don’t recommend: a roundup of recent critiques of non-recommended charities.
• Our take on microfinance:
  • 6 myths about microfinance, debunked.
  • “High repayment rates” don’t mean what you think.
  • Summary of our full 17-post series.
• Our take on other approaches to “economic empowerment”:
  • Agricultural training/technology/knowledge sharing
    
  • Gifts of livestock (e.g., Heifer International)
    
  • Social enterprise investment (e.g., Acumen Fund)
• 5-post series on “room for more funding”, the essential question about charities that no one seems to ask.
• Poor in the U.S. = rich: why U.S. poverty and overseas poverty are no comparison. (Also see our post on different definitions of “hunger”.)
• Some thoughts on GiveWell’s general approach:
  • We focus on gifts that help people, not gifts that feel good. We think the latter currently gets too much attention.
  • We don’t focus on “how much a charity spends on overhead.” A broad consensus is emerging that this is the worst way to pick a charity.
    
  • We believe that the burden of proof is on the charities to show that they’re having an impact. Our “default assumption” about a charity is that it isn’t living up to its story, and we think great charities are the exception rather than the rule.

  • As such, we are willing to give a charity a negative rating based on a failure to share meaningful information, even if we don’t have hard proof that it’s failing. This is probably the single most controversial aspect of our process, and we defend it at length in this post.

I believe that Philanthropedia has a promising model. We were glad to join with them on a recent press release urging donors to look beyond administrative expense ratios. We are potentially interested in collaborating with them and/or incorporating their work in the future. I examined their new report on microfinance with great interest.

At this point, I feel that Philanthropedia’s execution of the model has too many serious problems to make it of use to a donor, or to make it a positive influence on the sector. Philanthropedia is a very young organization and we expect/hope to see major improvement, which is why we are sharing these detailed thoughts (we have also discussed them with Philanthropedia).

Overview:

• The definition of a Philanthropedia “expert” is unclear, and we are concerned that the distinction between experience and expertise is being neglected.
• Only 31 of 131 “experts” even appear to have their names accessible on the site. The little we know about the “experts” gives us substantial concerns about their representativeness and credibility.
• Because of concerns about who the experts are, how they are chosen, and what criteria they are applying, we find that we can’t make sense of the final output (i.e., recommended microfinance organizations). We disagree strongly with the recommendations, but cannot see enough of what went into them to examine the sources of the disagreement.
• We would not want donors to give based on this report, not only because of the problems listed above, but because we believe that doing so would reinforce bad incentives.

Details follow. This is a long post, but if you are a nonprofit professional interested in the Philanthropedia model – or a donor considering following its recommendations – I urge you to read it all. There are more substantial concerns than I can present briefly.

---

The definition of a Philanthropedia “expert” is unclear

How does Philanthropedia define an “expert”? The most specific answer I can find to this question is via the Philanthropedia blog: “To find these experts, we do a variety of things such as research thought leaders in the space, look for program officers at foundations who specialize in this area, research universities who have faculty focused on these issues, identify journalists who write extensively about the topic, look for executive directors or heads of nonprofits working in the space, etc.” The Philanthropedia FAQ adds that “We target experts in a social cause through a combination of cold emails and warm referrals (on the basis of professional and personal connections).”

But the numerical breakdown of experts cites 8% academics, 8% funders, 53% nonprofit executives, and 24% unspecified “others.” (We’re not sure why the percentages add up to <100% but consider this a minor issue.)

• Why such a large proportion of nonprofit executives and “others”? Is this proportion deliberate, based on any particular view of the right proportions, or has it simply “fallen out” of which people Philanthropedia was able to reach and which chose to participate? Is it a function of the “personal connections” mentioned by the Philanthropedia FAQ?
• It seems clear that these numbers don’t come close to the total numbers of academics, nonprofit executives, etc. who would have relevant perspectives. So why these “experts” and not others? Philanthropedia’s description of its process is too vague to give an answer to this question.
• Who are the “others?” Are they all “journalists who write extensively on the topic” or do many fall into the “etc.” category?

Philanthropedia is all about trust. It provides very little information that donors can assess for themselves – the premise is that experts are better positioned to make recommendations. So it is essential to be crystal clear about just what constitutes the “expertise” that makes someone’s opinion so much more valuable and credible than an individual donor’s. We feel that Philanthropedia needs to add a substantial amount of information before it can be considered clear enough.

And as we have written before, we believe there is a huge difference between “experience” and “expertise” – particularly in the nonprofit sector with its broken feedback loops.

Who are the experts?

When we look at what information is available about the “experts” behind the microfinance report, we note:

• The report appears to have surveyed 131 total “experts.” As mentioned above, the vast majority of these are nonprofit executives and “other” and only 8% (~10) are academics.
• This leaves very little room for representation of microfinance skeptics. Nearly everyone, if not everyone, surveyed is likely coming from a worldview that embraces the core assumptions behind the pro-microfinance mentality, and thus embraces microfinance the way it is currently carried out. It seems fairly clear how this issue can lead to bias: for example, people who are already committed enough to microfinance’s potential to be funders/nonprofit executives seem likely to focus more on scale (reaching as many people as possible) rather than social impact (asking critical questions about microfinance’s effect on clients).
• Names and affiliations are provided for only 31 of the 131 “experts.” For 21 of the 31, the only information given is the name, position and organization name (and not all of these organizations can easily be found on the web – for example, is this the WAVE Foundation that employs Anwar Hossain, who is listed as one of the “experts”?)
• What we can see about these “experts” sheds very little light on how they were chosen and how representative they are.
  • Some are top-level executives and some are not.
  • 3 of the 31 represent Unitus; one represents Tearfund, a very large charity working in diverse areas; but other major U.S. microfinance-focused charities such as Grameen Foundation and FINCA have no representatives in this set. (Perhaps Unitus’s heavy representation has something to do with its low rank, since Philanthropedia requires that nonprofit professionals not vote for their own organization.)
  • In what regions have these “experts” done their work? With what kinds of programs? In what ways might their experience and allegiances be biased? Do they have valuable experience and impressive accomplishments or merely many years of being employed in this area? We don’t have the information that could address these questions.

The recommended charities

As stated above, Philanthropedia provides very little information on the recommended charities aside from their names, links to their websites, and figures whose exact meaning is unclear but which seem to indicate the level of “expert support.”

Our main observation is that the list reads like a “who’s who in large U.S. microfinance charities.” This pertains to our above observation that microfinance skeptics don’t seem well represented.

Most of the ratings are clustered fairly close together (7%-11% range). Accion is at the top with 19%, and CARE, FINCA and Unitus are at the bottom. It’s unclear exactly what these numbers mean – should we take from this that Accion is particularly outstanding or that Unitus has problems? Probably not, since Philanthropedia states that it doesn’t rank nonprofits.

We would not, at this point, recommend any of the charities listed:

• Accion, Grameen, Opportunity International, Women’s World Banking and Unitus all fall into the category of “consultant” charities that we’ve expressed reservations about. For all of these organizations, we have serious concerns about their social-impact-focused due diligence.
• Freedom from Hunger, BRAC and CARE all run a wide variety of programs including but not limited to microfinance. It’s not clear that your donation to these organizations will even fund expansion of microfinance (regardless of whether you restrict it).
• We know less about Pro Mujer and FINCA. We have not looked deeply into either as part of our ongoing grant application (Pro Mujer does not work in Africa and FINCA declined to apply).
• MIX Market is an information source (and an excellent one, in my view), not a provider of direct services. I would be open to recommending it but would need to know what it would do with more funds.
• As for Kiva, we have written a lot about it. At this point our basic take is that it’s effectively getting your money to partner microfinance institutions, and we don’t have enough information, or confidence in, its due diligence process on partners from a social impact point of view (our posts on Kiva list a couple particular, substantial points of concern).

The “expert quotes” provided by Philanthropedia do not address these concerns, and in fact raise questions about whether the “experts” are even trying to help donors accomplish good. Take the Kiva page for an example. (Click “Expert assessment” from that link to view quotes.) Nearly all strengths listed pertain to Kiva’s marketing ability: “They have a cute model,” “They are appealing to donors,” and more. In other words, you are being advised to give to a charity because it’s good at making you want to give to it.

The few comments that pertain to other aspects of Kiva raise other concerns:

• “They are truly innovative as a capital source for microfinance institutions.” The exact nature of the innovation is unspecified, but probably amounts to more praise for the marketing.
• “Kiva charges 0% interest rate on capital.” Kiva charges 0% to its partners, but borrowers have to pay interest. Is that a good thing, and if so, why?
• “They provide a direct link from funder to borrower” – this is simply false.
• “They are not neglecting the US domestic poverty issue” – this comment comes up in more than one organization’s profile. We have argued that it is better not to focus on US domestic poverty; in any case, this is clearly a difficult, value-laden judgment call.

The “expert quotes” on other organizations raise similar concerns. Overall, it is not apparent that the “experts” are giving advice based on how an individual donor can have the most impact. It is not clear what their recommendations mean.

As for our own take on which microfinance charities are best, we have deep concerns about this sector that have led us to go beyond U.S. charities and seek out outstanding microfinance institutions. We’ll be writing about more about the best we’ve found, the Small Enterprise Foundation.

We are coming from an attitude of skepticism toward microfinance in general. Our skepticism is certainly debatable, but we have been explicit about it and discussed our thoughts on microfinance at length. A donor who wants to know where our advice is coming from and what it’s based on can find out. We feel it is important that donors have similar context on the people cited by Philanthropedia.

Reinforcing bad incentives

We pay a lot of attention to incentives. When discussing how we should rate charities, we constantly ask, “If charities all knew we were handing out ratings on this basis and expected donors to give based on our ratings, how would they work the system? What would we expect to change?” (For example, we give negative reviews to charities that publish no information, not just charities whose materials raise concerns; we don’t want to provide incentives to hide information.)

We feel that if Philanthropedia stuck to its current approach and donors used it:

• “Experts” would not have any incentives to increase their own transparency above current, unacceptable levels. They could influence others’ giving by giving recommendations without substantive reasons and without even disclosing their names.
• Charities would have strong incentives to do whatever got them recommended by Philanthropedia’s set of experts. This could include networking, flattery, wining/dining, outright conflicts of interest, and learning to speak of their programs in ways that would appeal to Philanthropedia experts, all of which might end up being more cost-effective from the charities’ perspective than optimizing and demonstrating their social impact.

Bottom line
I urge Philanthropedia to:

• Publish exhaustive details of how “experts” are defined, selected and invited.
• Publish names and affiliations of all experts who are invited to participate, whether or not they accept.
• It is particularly important to publish names and affiliations for those who do participate because donors should not be asked to trust anonymous people.
• Address the concerns we have raised about the representativeness and credibility of experts, if these concerns still appear valid once the identities and process are disclosed.
• Publish the templates for all surveys sent to experts. Be explicit about the criteria by which experts are asked to recommend charities.
• Ask experts to make their best case publicly and to be clear about whether they’re recommending charities to impact-focused donors, praising charities for marketing abilities, or something else.

We believe the Philanthropedia model has a lot of potential. We would find great use, for donors and for ourselves, in a site that could credibly claim to have assembled the charities with the most support from people with relevant experience.

Philanthropedia has accomplished a lot in a short time, but we don’t believe its content is yet appropriate for distribution to individual donors looking for help with their giving decisions.