I believe that Philanthropedia has a promising model. We were glad to join with them on a recent press release urging donors to look beyond administrative expense ratios. We are potentially interested in collaborating with them and/or incorporating their work in the future. I examined their new report on microfinance with great interest.
At this point, I feel that Philanthropedia’s execution of the model has too many serious problems to make it of use to a donor, or to make it a positive influence on the sector. Philanthropedia is a very young organization and we expect/hope to see major improvement, which is why we are sharing these detailed thoughts (we have also discussed them with Philanthropedia).
- The definition of a Philanthropedia “expert” is unclear, and we are concerned that the distinction between experience and expertise is being neglected.
- Only 31 of 131 “experts” even appear to have their names accessible on the site. The little we know about the “experts” gives us substantial concerns about their representativeness and credibility.
- Because of concerns about who the experts are, how they are chosen, and what criteria they are applying, we find that we can’t make sense of the final output (i.e., recommended microfinance organizations). We disagree strongly with the recommendations, but cannot see enough of what went into them to examine the sources of the disagreement.
- We would not want donors to give based on this report, not only because of the problems listed above, but because we believe that doing so would reinforce bad incentives.
Details follow. This is a long post, but if you are a nonprofit professional interested in the Philanthropedia model – or a donor considering following its recommendations – I urge you to read it all. There are more substantial concerns than I can present briefly.
The definition of a Philanthropedia “expert” is unclear
How does Philanthropedia define an “expert”? The most specific answer I can find to this question is via the Philanthropedia blog: “To find these experts, we do a variety of things such as research thought leaders in the space, look for program officers at foundations who specialize in this area, research universities who have faculty focused on these issues, identify journalists who write extensively about the topic, look for executive directors or heads of nonprofits working in the space, etc.” The Philanthropedia FAQ adds that “We target experts in a social cause through a combination of cold emails and warm referrals (on the basis of professional and personal connections).”
But the numerical breakdown of experts cites 8% academics, 8% funders, 53% nonprofit executives, and 24% unspecified “others.” (We’re not sure why the percentages add up to <100% but consider this a minor issue.)
- Why such a large proportion of nonprofit executives and “others”? Is this proportion deliberate, based on any particular view of the right proportions, or has it simply “fallen out” of which people Philanthropedia was able to reach and which chose to participate? Is it a function of the “personal connections” mentioned by the Philanthropedia FAQ?
- It seems clear that these numbers don’t come close to the total numbers of academics, nonprofit executives, etc. who would have relevant perspectives. So why these “experts” and not others? Philanthropedia’s description of its process is too vague to give an answer to this question.
- Who are the “others?” Are they all “journalists who write extensively on the topic” or do many fall into the “etc.” category?
Philanthropedia is all about trust. It provides very little information that donors can assess for themselves – the premise is that experts are better positioned to make recommendations. So it is essential to be crystal clear about just what constitutes the “expertise” that makes someone’s opinion so much more valuable and credible than an individual donor’s. We feel that Philanthropedia needs to add a substantial amount of information before it can be considered clear enough.
And as we have written before, we believe there is a huge difference between “experience” and “expertise” – particularly in the nonprofit sector with its broken feedback loops.
Who are the experts?
When we look at what information is available about the “experts” behind the microfinance report, we note:
- The report appears to have surveyed 131 total “experts.” As mentioned above, the vast majority of these are nonprofit executives and “other” and only 8% (~10) are academics.
- This leaves very little room for representation of microfinance skeptics. Nearly everyone, if not everyone, surveyed is likely coming from a worldview that embraces the core assumptions behind the pro-microfinance mentality, and thus embraces microfinance the way it is currently carried out. It seems fairly clear how this issue can lead to bias: for example, people who are already committed enough to microfinance’s potential to be funders/nonprofit executives seem likely to focus more on scale (reaching as many people as possible) rather than social impact (asking critical questions about microfinance’s effect on clients).
- Names and affiliations are provided for only 31 of the 131 “experts.” For 21 of the 31, the only information given is the name, position and organization name (and not all of these organizations can easily be found on the web – for example, is this the WAVE Foundation that employs Anwar Hossain, who is listed as one of the “experts”?)
- What we can see about these “experts” sheds very little light on how they were chosen and how representative they are.
- Some are top-level executives and some are not.
- 3 of the 31 represent Unitus; one represents Tearfund, a very large charity working in diverse areas; but other major U.S. microfinance-focused charities such as Grameen Foundation and FINCA have no representatives in this set. (Perhaps Unitus’s heavy representation has something to do with its low rank, since Philanthropedia requires that nonprofit professionals not vote for their own organization.)
- In what regions have these “experts” done their work? With what kinds of programs? In what ways might their experience and allegiances be biased? Do they have valuable experience and impressive accomplishments or merely many years of being employed in this area? We don’t have the information that could address these questions.
The recommended charities
As stated above, Philanthropedia provides very little information on the recommended charities aside from their names, links to their websites, and figures whose exact meaning is unclear but which seem to indicate the level of “expert support.”
Our main observation is that the list reads like a “who’s who in large U.S. microfinance charities.” This pertains to our above observation that microfinance skeptics don’t seem well represented.
Most of the ratings are clustered fairly close together (7%-11% range). Accion is at the top with 19%, and CARE, FINCA and Unitus are at the bottom. It’s unclear exactly what these numbers mean – should we take from this that Accion is particularly outstanding or that Unitus has problems? Probably not, since Philanthropedia states that it doesn’t rank nonprofits.
We would not, at this point, recommend any of the charities listed:
- Accion, Grameen, Opportunity International, Women’s World Banking and Unitus all fall into the category of “consultant” charities that we’ve expressed reservations about. For all of these organizations, we have serious concerns about their social-impact-focused due diligence.
- Freedom from Hunger, BRAC and CARE all run a wide variety of programs including but not limited to microfinance. It’s not clear that your donation to these organizations will even fund expansion of microfinance (regardless of whether you restrict it).
- We know less about Pro Mujer and FINCA. We have not looked deeply into either as part of our ongoing grant application (Pro Mujer does not work in Africa and FINCA declined to apply).
- MIX Market is an information source (and an excellent one, in my view), not a provider of direct services. I would be open to recommending it but would need to know what it would do with more funds.
- As for Kiva, we have written a lot about it. At this point our basic take is that it’s effectively getting your money to partner microfinance institutions, and we don’t have enough information, or confidence in, its due diligence process on partners from a social impact point of view (our posts on Kiva list a couple particular, substantial points of concern).
The “expert quotes” provided by Philanthropedia do not address these concerns, and in fact raise questions about whether the “experts” are even trying to help donors accomplish good. Take the Kiva page for an example. (Click “Expert assessment” from that link to view quotes.) Nearly all strengths listed pertain to Kiva’s marketing ability: “They have a cute model,” “They are appealing to donors,” and more. In other words, you are being advised to give to a charity because it’s good at making you want to give to it.
The few comments that pertain to other aspects of Kiva raise other concerns:
- “They are truly innovative as a capital source for microfinance institutions.” The exact nature of the innovation is unspecified, but probably amounts to more praise for the marketing.
- “Kiva charges 0% interest rate on capital.” Kiva charges 0% to its partners, but borrowers have to pay interest. Is that a good thing, and if so, why?
- “They provide a direct link from funder to borrower” – this is simply false.
- “They are not neglecting the US domestic poverty issue” – this comment comes up in more than one organization’s profile. We have argued that it is better not to focus on US domestic poverty; in any case, this is clearly a difficult, value-laden judgment call.
The “expert quotes” on other organizations raise similar concerns. Overall, it is not apparent that the “experts” are giving advice based on how an individual donor can have the most impact. It is not clear what their recommendations mean.
As for our own take on which microfinance charities are best, we have deep concerns about this sector that have led us to go beyond U.S. charities and seek out outstanding microfinance institutions. We’ll be writing about more about the best we’ve found, the Small Enterprise Foundation.
We are coming from an attitude of skepticism toward microfinance in general. Our skepticism is certainly debatable, but we have been explicit about it and discussed our thoughts on microfinance at length. A donor who wants to know where our advice is coming from and what it’s based on can find out. We feel it is important that donors have similar context on the people cited by Philanthropedia.
Reinforcing bad incentives
We pay a lot of attention to incentives. When discussing how we should rate charities, we constantly ask, “If charities all knew we were handing out ratings on this basis and expected donors to give based on our ratings, how would they work the system? What would we expect to change?” (For example, we give negative reviews to charities that publish no information, not just charities whose materials raise concerns; we don’t want to provide incentives to hide information.)
We feel that if Philanthropedia stuck to its current approach and donors used it:
- “Experts” would not have any incentives to increase their own transparency above current, unacceptable levels. They could influence others’ giving by giving recommendations without substantive reasons and without even disclosing their names.
- Charities would have strong incentives to do whatever got them recommended by Philanthropedia’s set of experts. This could include networking, flattery, wining/dining, outright conflicts of interest, and learning to speak of their programs in ways that would appeal to Philanthropedia experts, all of which might end up being more cost-effective from the charities’ perspective than optimizing and demonstrating their social impact.
I urge Philanthropedia to:
- Publish exhaustive details of how “experts” are defined, selected and invited.
- Publish names and affiliations of all experts who are invited to participate, whether or not they accept.
- It is particularly important to publish names and affiliations for those who do participate because donors should not be asked to trust anonymous people.
- Address the concerns we have raised about the representativeness and credibility of experts, if these concerns still appear valid once the identities and process are disclosed.
- Publish the templates for all surveys sent to experts. Be explicit about the criteria by which experts are asked to recommend charities.
- Ask experts to make their best case publicly and to be clear about whether they’re recommending charities to impact-focused donors, praising charities for marketing abilities, or something else.
We believe the Philanthropedia model has a lot of potential. We would find great use, for donors and for ourselves, in a site that could credibly claim to have assembled the charities with the most support from people with relevant experience.
Philanthropedia has accomplished a lot in a short time, but we don’t believe its content is yet appropriate for distribution to individual donors looking for help with their giving decisions.
I am big believer in the potential of Philanthropedia, but you really nailed the concerns about their model that I have. And the “reinforcing bad incentives” section made points that I hadn’t considered before but should have.
That said, I’m not sure the microfinance section of your critique is that on point in that I don’t think Philanthropedia is trying to answer the question of whether a “sector” as a whole is having an impact, whether that sector is “microfinance”, “climate change”, or the other sectors they cover. Instead, they’re answering a narrower question: “I want to give in a certain area: which non-profits in that sector are doing the best work”.
And while I think it’s vitally important that someone, whether it is GiveWell or others, explore the macro question of whether the sector as a whole is having an impact, I don’t think it’s illegitimate for Philanthropedia not to address that issue. In fact, given their model, I think they are right to leave it alone and focus just on identifying the tops orgs in different fields. It’s what their model is good for.
Holden, thank you again for taking a look at microfinance, I find the report to be very interesting and insightful. I have posted the first part of my response here: http://blog.myphilanthropedia.org/?p=41. I will be adding more specific comments in Part 2 tomorrow. I look forward to hearing your thoughts and continuing the discussion!
Gordon, you are making a good point. My personal opinion is that we have a responsibility to inform donors about the status quo in different social causes. That’s why in the future I envision adding more information, perhaps a page, for each social cause that can educate the donor on the important issues. I touch on this in Part 2 (which is not yet published).
Thanks all for the comments, keep them coming, I look forward to learning more!
When I first took a look at Philanthropedia’s web site a few months ago, I had questions that reflect all the concerns you are outlining. Who are the exerts? What is their methodology? Why so little focus on outcomes despite Philanthropedia’s expressed aim to evaluate “effectiveness” and “nonprofit impact?”
I have read Philanthropedia’s response (part 1), and although I like the idea of Philanthropedia and understand the project is evolving, I tend to agree with Holden that sharing the content seems premature. The foundation for the recommendations is just too unclear for donors to make informed decisions.
I have now posted the second part of my response here: http://blog.myphilanthropedia.org. Sorry for the slight delay!
Ingvild, thank you for sharing your perspective. I personally disagree with it, because keeping the research results private would promote the wrong incentives (especially in helping us improve as well as the other elements that I wrote about in my response). I invite you to share specific criticisms or suggestions for improvement too!
Gordon, we feel that the answer to the “narrow question” you raise is heavily dependent on the attitude toward the “macro question.” Our stance on microfinance is a case in point. We feel that the sector as a whole has major problems, so we encourage donors to go in unconventional directions like directly funding the small, non-US-registered Small Enterprise Foundation. By contrast, a group of insiders within the U.S. microfinance charity circuit would be expected to point toward the current “big names” in the field (as appears to have happened).
Comments are closed.