The GiveWell Blog

Microfinance: The multi-billion-dollar aid sector that we’re just starting to learn about

(updated on: ) | by Holden

GiveWell attended the Microfinance Innovation & Impact Conference via Board member Tim Ogden and part-time employee Stephanie Wykstra. (Our full-time staff could not attend as we are all in India). Our main takeaways:

  • The people involved with this conference, and the nonprofits working with them, are producing a lot of promising, interesting, rigorous and informative work on how microfinance affects the poor and how these effects might be improved.
  • While microlending is the form of microfinance that has been most funded, celebrated and scaled up over the decades, it appears to be the least proven and least-well understood in terms of its impact on the poor. In many ways it seems that good information on how (and to what extent) it’s helpful is just starting to become available.

Content presented at the conference
There is much abstract agreement about the importance of learning from good research, but we feel that there are few concrete examples of groups actually doing it. A high concentration of those examples could be found at this conference. We have written before about how impressed we are with the work of the Poverty Action Lab and Innovations for Poverty Action (groups behind this conference), and that continues to be the case.

The research has already been summarized in a large number of posts by others (see the roundup by GiveWell Board member Tim Ogden). Our quick take:

Microsavings, microinsurance, and “ultrapoor programs” are clearly promising. High-quality studies found significant improvements on metrics like income, food security, and children’s ability to attend school when these programs – which differ markedly from the loans most people associate with “microfinance” – were rolled out. Programs included accounts intended to help people pre-commit to saving (more); intensive promotion of difficult-to-sell insurance products (more), and “ultra-poor programs” that target people with extremely low incomes and offer direct benefits such as cash/asset transfers and education, rather than financial products (more).

Unfortunately, we know of no clear vehicles through which individual donors can fund these sorts of programs. We’ve written before about the frustrating situation of seeing a good program without a good funding vehicle, and called for major funders to consider this problem. There do appear to be similarities between the “ultrapoor program” and the work of GiveWell-recommended charity Village Enterprise Fund.

The impact of microlending is, at this point, not very well understood. A new high-quality study in Morocco found no positive social impact overall, though it did find that people with existing agricultural businesses appeared to invest more/differently in their businesses (more). These results were similar to those observed in the only previous high-quality study of traditional microlending, which took place in a very different environment (Hyderabad, India) and was summarized last year by David Roodman.

There was significant discussion of the risks that people might be overborrowing (something we are quite concerned about, and something that there is apparently still little information to assess). And new data from the Philippines reinforced a point we have emphasized in the past – that microloans are more likely to finance consumption, and less likely to finance entrepreneurial investment, than most donors suppose. From Philanthropy Action’s summary:

Around 46 percent of clients used new debt to pay down old debt, and 28 percent used funds for a single large household purchase. In all, 39 percent of the money goes unaccounted for, 15 percent is used to pay down other loans, 9 percent goes into the household and 37 percent is invested in the business. Curiously, Karlan’s study got at these results by asking clients the same questions in a number of different ways. Not surprisingly, when the banks ask clients why they want the funds only 2 percent of clients admit potential uses other than the business, even after the funds are given. Only when an independent surveyor asks indirectly do any relevant percentage of people admit to non-entrepreneurial uses for credit.

Studies are finding interesting potential ways to improve the impact of microlending. One possibility is better targeting (as discussed above, microlending appears to affect different people differently). Two other possibilities raised at the conference, both with encouraging preliminary evidence behind them, were the implementation of grace periods for repayment (more) and financial education focusing on “rules of thumb” rather than formal accounting (more).

Learning how microlending works – better late than never
It appears to us that an enormous amount of philanthropic capital has gone into microfinance. Last year we looked at data from 2008 and found that $11.7 billion had gone to microfinance funding of some kind, with $2.2 billion being grants. For context, total official aid earmarked for health was about $3 billion disbursed / $8 billion committed in the same year (data from Aid Data).

My greatest fear about microfinance is that all (or nearly all) of that money is chasing a good story rather than a good program.

In this context, it’s hard for us to see how one can argue that good evaluation is too expensive. The total budget of Innovations for Poverty Action that year was just under $6 million (data from NCCS).

The process of giving: A personal story (guest post from Ian Turner)

(updated on: ) | by Guest

This is a guest post from Ian Turner about how he decided what charity to support for his most recent donation. Ian and GiveWell staff had several in-depth conversations as he worked through his decision, so we invited him to share his thought process here. Note that GiveWell has made minor editing suggestions for this post (though Ian determined the final content).

Every person who gives to charity does so for a different reason, and these reasons are as numerous and as varied as the thousands of causes and millions of charities that pursue them. I’d like to share a little about recent choices I’ve made, and how that approach to giving has developed over time.

For me, the essence of giving comes down to three distinct beliefs or realizations. These are, in turn, the limitations of money; the importance of money; and an actual desire to help others. When you combine these three beliefs with actual resources to give, one inevitably must give and must give for impact – that is, one must give well. Thus my approach. I’m not attached to any particular sector, cause, approach, country, or region. But I do look to maximize the effect of my gifts on the human condition overall.

First gifts: A rough start

In the very beginning, as a child, my father greatly influenced my thoughts about giving. He placed ideas in my head such as “don’t give to panhandlers directly, give to organizations”, and created the legitimacy, if not the expectation, of giving freely. Out of this idea, the desire to help now (and not in the unpredictable future), and out of the tradition of tithing shared by many religions, I decided even before I ever had any real earnings that I would give 10% of income (after taxes) to charity, on an ongoing basis. That is a practice that I have continued in the over a decade since.

I had no idea at the time how hard aid actually is, how unsubstantiated most of the nonprofit sector is, or how hard it is to find an effective organization to give to. My first gift was in September 1999, in the amount of $650 (10% of the income from my first summer job) to an organization picked more or less at random: The Unitarian Universalist Service Committee (UUSC). I didn’t have a clear objective other than improving the human condition generally, for which their mission seemed broad enough, and the only real thing that set them apart is that they made the right ask at the right time.

I was disappointed to receive, over the following year, a variety of communications from the UUSC, indicating that my money was being spent in all kinds of ways that (to me) didn’t make sense, from US political activism to questionable (to me) small-scale village business projects. I resolved that I would do a little better next time.

So the next year, after another (more profitable) summer job, I made a different gift. At the time I figured world hunger was a bigger problem than fair trade for cocoa farmers, but thanks to seeds planted by my father I figured there was a better way to tackle hunger than just handing out food. And that’s how I found Freedom from Hunger. Freedom from Hunger operates a large variety of international programs, the centerpiece of which claims to combine microcredit and education for poor rural women.

To evaluate the organization, my main source of information was annual reports. I went back a few years, but didn’t look at any studies or third-party information. Even with this little information or experience, however, I could tell that there were some differences between organizations and looked at a few different ones. Freedom from Hunger had what seemed to be the most impressive annual reports, which I think means intelligent copy that told a story which seemed reasonable, at least from my naïve perspective. So in 2000 I gave them $2,500.

As an exercise, I recently revisited Freedom from Hunger using my now more experienced perspective, to see what evidence of impact they really had. The answer could be described as more than most, and less than the best. Freedom from Hunger did perform a number of controlled studies examining different aspects of its flagship programs, but the results seem questionable, prone to cherry-picking, and it’s not clear to what degree the evaluated treatments match the programs actually implemented worldwide. Freedom from Hunger’s programs today are also significantly more complex than they were in 2000.

One of the biggest lessons I learned from the 2000 gift is that giving well is hard, harder than I ever would have guessed two years previous. I decided I could manage the process better if I made larger gifts less frequently. Eventually, I settled on keeping 10% of my income in a special account each year and then only actually giving it away triennially. As it turns out, I didn’t have any significant income again until 2004, and for tax reasons my next gift was not until 2006.

As an aside, I should note that the triennial 10% gift is specifically for philanthropic efforts. I also make gifts to my church, to people I know personally who are in need, and to political organizations whose causes I support. But I don’t consider that philanthropy, I don’t make the analysis that I would for philanthropic gifts (which are much larger), and I don’t count such gifts toward my 10% donation.

Seeking a better way

After 2000 I did not give again until 2006, but I was not idle during the intervening years. Through The Economist I learned about two organizations of significance, Pratham (Economist article) and Geneva Global (Economist article). Pratham is an Indian educational charity notable for its built-in evaluation approach and closed-loop feedback process. Although Givewell did not exist at the time I discovered Pratham, it has since been reviewed and recommended by Givewell. Geneva Global is a for-profit company that provides advice to donors, especially in terms of impact prediction and evaluation. For their services they charge a fee equal to 15% of the amount donated. Of course, when the time came I also had another look at Freedom from Hunger. I didn’t have any other way to cut through the million charity options, so those became by default my three options to evaluate.

Of the three, I eliminated Freedom from Hunger early on. By now I saw that there was more to impact than glossy annual reports, and I had a growing skepticism about microfinance generally. I was still looking to address root causes; education seemed like a better bet; and Pratham seemed to be making a big difference there. Geneva Global’s main advantage was an appeal to expertise: Their job was to know where to give, so the only question in my mind was whether their expertise was worth the 15% price tag.

In the end, the choice came down to my own procrastination: I didn’t start contacting organizations to ask questions until December, and by the time I found out who at Pratham could answer my questions, they were already away for the holidays. I couldn’t get any substantive information from Pratham until after the new year – not even basic financial information, let alone detailed information about programs. Geneva Global, on the other hand, was beyond responsive: They had lots of detailed information on specific projects, and best of all, a brand-new fund which was offering a 1:1 match to attract starting capital. So instead of paying 15% overhead for Geneva Global, I would end up getting a 70% uplift. In the end, on December 29, 2010, I put a check in the mail for $14,000 toward Geneva Global’s Rwanda Human Empowerment Fund, a collection of projects focused on education, health, and land rights in Rwanda.

As it turns out, I probably chose poorly in 2006. Geneva Global went through a few different purchases and restructurings before winding down the Fund and sending out a final report. This indicated that the Fund had only attracted about a third of the expected amount, but that Geneva Global had begun spending in the first year on the assumption that additional donations would continue to come in. As a result most projects were cut off or could not be continued in the second and third years of the fund’s operation.

In addition, some projects looked highly questionable to my untrained eye. For example, one school project started construction on new classroms, but then shifted midway to make the building a large $64,000 dining hall instead. The school already had a dining hall; the only benefit mentioned in the report was that all 1,000 students could dine at once rather than in shifts. Another project complained that a grantee “lacks professionalism, transparency, and honesty”, leaving the project unfinished and the funds missing. Roughly 1/3 of the total fund was invested in a land rights initative, but the report diplomatically notes that the grantee, the International Justice Mission, “faced many challenges as an outsider”. Finally, none of the projects seemed to have much in the way of quantitative evaluation; with respect to the health initative, the executive summary notes that “Though no formal study has been conducted on the impact of these improved services on infant and maternal mortality rates, we have no doubt that they consequently declined”.

My overall impression was that Geneva Global’s in-county contractor had been given some combination of too much leeway and too little training, and that a substantial amount of the money was lost to subcontractors, spent on unplanned expenses, or wasted on poorly planned projects.

The main lesson I took away from this is that even with a fair amount of effort, giving for impact is really hard! So for my next gift, made in 2009, I determined to do a better job. I started the process early in the year with only Pratham in hand, but quickly discovered resources like GiveWell and Good Intentions are Not Enough. It seems that over the course of the 2000s, a lot of people got interested in impact philanthropy all at once.

Information: A trickle becomes a torrent

GiveWell turned out to be a critical source of information for my 2009 donation. Besides pointing me to exceptional organizations like Population Services International (PSI; Givewell review) and VillageReach (Givewell review), the GiveWell blog and its archives provided a steady supply of new perspectives on charity and giving, ideas which I had never encountered before. Sure, I had long suspected that there were large differences between charities, and had discovered through my own mistakes that giving was hard, but I had never thought about things like offsetting impacts, the complexities of microfinance interest and repayment rates, the costs of receiving aid, or room for more funding. GiveWell suggested some great organizations and then gave some useful benchmarks by which to measure them.

In the end, I settled on a shortlist of Pratham, PSI, and VillageReach. The latter two I discovered through GiveWell. PSI is a large international health charity with some innovative ideas about how to leverage markets to improve distribution and use of health products and services like condoms, bed nets, circumcision, and vaccination. VillageReach is a US charity originally operating exclusively in Mozambique which is notable for having one of the most complete and inarguable demonstrations of impact among charities of any size.

GiveWell also recommended a few other organizations that didn’t pique my interest as much. The Against Malaria Foundation (AMF; Givewell review) interested me less than PSI primarily because of the latter’s inclusion of reproductive health as an essential part of health care delivery. And the Stop TB Partnership (Givewell Review) is an organization which seems to be very effective, but which I heard about too late to consider for a 2006 donation. In addition, giving to the Stop TB partnership would be significantly more difficult, since it is not a US organization and since my 2009 donation was mostly in the form of appreciated stock.

I contacted all three organizations (Pratham, PSI, and VillageReach) and engaged each of them with a battery of questions regarding their programs, evaluations, use of marginal funds, and much more. Owing to the earlier date (and perhaps larger donation size), I had a very different experience compared to 2006: All three organizations were very much forthcoming with both information and time. I read a lot of reports, came back with more questions, and spent hours on the phone, both with charities and with folks from GiveWell. This was a tough decision, because the organizations are fairly different from each other but each with a compelling case.

Pratham was the first to be disqualified. Essentially, the majority of Pratham’s spending goes to two types of programs: Field programs, where Pratham operates remedial education sessions directly on a village-by-village basis, and Read India, where Pratham partners with local governments to improve the quantity and quality of education in public schools. In an attempt to reach the massive scale required for nationwide impact in India, Pratham has focused most of its growth on Read India. But the best case and evidence, both in terms of program design and in terms of third-party evaluation, is for the field programs. In late 2009, Pratham was in the midst of conducting rigorous controlled trials of a few Read India implementations. I have no doubt that they’ll make a strong showing for my 2012 gift, but to me there was not enough evidence on Read India to justify a gift in 2009.

That left PSI and VillageReach: Both health charities, though with very different scale and approaches, making for a difficult comparison. My decision was made more difficult by the fact that VillageReach was in the middle of its own strategic planning process while my evaluation was underway. Indeed, I had originally decided against VillageReach, but gave it new consideration according to new information provided at the last minute.

PSI is a large organization and a relatively known quantity. They provide a fairly advanced quantitative model attempting to translate any and all health interventions into Disability-Adjusted Life Years (DALYs), then allocating funds to the most cost-effective projects, as measured in cost per DALY. Of course, a large organization like PSI receives a lot of restricted funding, so the organization operates some projects even if they have very high cost per DALY. Nonetheless, PSI prepares detailed and easy-to-read reports on both impact and cost-effectiveness. The questions are likewise fairly straightforward: Is the model which translates pregnancies avoided into DALYs a correct one? Are the projects sustainable? Is PSI simply replacing health products and services that would be provided by free markets or governments, or is it expanding markets to consumers that would not be reached otherwise? PSI’s programs are clear but determining true impacts must unfortunately leave some of these questions unanswered.

VillageReach

VillageReach is another story. In order to understand the organization, you need to understand its history. VillageReach aims to improve health outcomes by making distribution systems for health products, especially vaccines, work more efficiently and effectively. The organization started its programs in 2002 with a pilot project in the Cabo Delgado province of Mozambique. VillageReach created a new approach to vaccine distribution, shifting the burden from health clinic workers to a set of staff and equipment dedicated to distribution. In addition to delivery personnel, VillageReach created a new job position to monitor the inventory of vaccines and the effectiveness of the delivery schemes. All these people nominally worked for the government, and all this equipment nominally belonged to the government, but VillageReach paid all the costs, whereas costs for pre-existing systems had been covered by the government. In order to make all this work, VillageReach had to create a whole new distribution company for propane gas, which was previously prohibitively expensive to purchase in Cabo Delgado. VillageReach reports that the company, named VidaGas, is now self-sustaining.

The pilot program ended in 2007, at which point VillageReach handed control over to the government, which promptly dismantled VillageReach’s work and returned to the previously extant approach. But in the meantime, VillageReach prepared two significant reports: One on impact and the other on cost. The two reports compared the cost and effectiveness of the VillageReach approach to the one the government had been using previously, by comparing vaccination rates in the trial province of Cabo Delgado to those in the neighboring province of Niassa. The conclusion is as powerful as it is inescapable: When compared to its neigbor, Cabo Delgado saw much higher rates of vaccination at significantly lower total cost. In other words, the VillageReach approach was lower-cost, not only on a per-vaccination basis but on an overall basis. Indeed, the vaccination program was so cheap that as a donor it would be considered a bargain even considering that VillageReach had to displace all the government spending.

This is the history I had in hand when I started talking to VillageReach in 2009. And when I asked the question, “what’s next?”, I was concerned about the answer. Of course, the vaccination program in Cabo Delgado had already been shut down for two years, and in its place VillageReach talked about a variety of consulting and advocacy programs. But I had no evidence about VillageReach’s effectiveness as council in any country, let alone in those far away from Mozambique. Essentially, given the future plan VillageReach had presented to me, I had to reject them: All the evidence of effectiveness was in relation to a program that was no longer operating.

So I decided to give to PSI. But I didn’t want to give them a free pass, so I resolved to evaluate them as thoroughly as I could, quickly compare that to some other organizations’ claims, and if PSI seemed adequate to choose them.

But, at the last minute, due in part to urging by GiveWell, I gave VillageReach one last phone call, wherein they explained a new agreement with the government of Mozambique to roll out the Cabo Delgado vaccination distribution approach nationwide. This seemed to me to be much closer to VillageReach’s proven pilot project and thus much likelier to be a success. The pilot project had been so successful that I couldn’t say no to a nationwide deployment, and so I gave my $35,000 2009 contribution to VillageReach. I later found out, much to my surprise, than this gift was and is their largest ever donation from an individual.

Back to the Future

Which brings us to the end of the story (for now). In an adventure spanning a bit over a decade, I’ve learned a lot of surprising things: Doing good is harder than it looks. Good intentions are not enough. Helping others is rewarding. And maybe, with the careful application of analytic thought, giving money really can make a difference. Like science, it’s a process of discovery. You try something out, you look to see how it goes, you learn something, and then you repeat the process, and hopefully your little extra knowledge and experience lets you see something deeper the next time around. It’s not easy, but it is possible.

As I look forward to the future of philanthropy, both mine and others’, it’s an exciting picture. There will be new discoveries about the nature of giving, and as the nonprofit sector grows daily more interested in measuring impact, the number of opportunities to give effectively grows with it. And I’m confident that my own skills at giving will continue to grow as well. Charity can change the world: Just make sure you look before you give.

Acknowledgements

This essay would not have been possible without the thoughts, ideas, inspiration, proofreading, and support of a surprisingly large group of friends and family and of the GiveWell staff. You have provided me with essential feedback for which I am unremittingly grateful.

Thanks also to the many people and organizations that have provided me with time, information and ideas throughout my philanthropic career. A donor can accomplish nothing without a place to give, and I am grateful for each of you, even when I have directed funds elsewhere. Our field is in an interesting time at the moment, and is well-poised for significant new accomplishments. Let’s make it happen.

Engineers Without Borders earns “Notable” distinction

| by Holden

Engineers Without Borders is a group we’ve never encountered before, focused on “enabling rural Africans the opportunity to access clean water, generate an income from humble farms, and access critical infrastructure and services.” (Here’s a list of its projects in Africa.)

This group deserves recognition for doing the simplest thing guaranteed to improve GiveWell’s view of it: disclosing a failure.

Its 2009 and 2010 failure reports are publicly and prominently published. These aren’t “quasi-failures” along the lines of “Didn’t raise enough money for X”; they involve staff misreading their environments, both in terms of culture (poorly anticipating how a project’s timeline would be affected by holiday schedules) and in terms of interactions between aid institutions (nearly losing funding for a project by what reads to me like hurting a funder’s feelings). I encourage you to read the reports themselves rather than relying on our summary, since they are eye-opening.

Without knowing anything else about the work of Engineers Without Borders, we commend it for clearly and publicly discussing its own shortcomings, and thereby creating more public information on the complexities of aid.

Hat tip: Aid Watch

LAPO: Case study on due diligence by microfinance funders

(updated on: ) | by Holden

Updated 10/19/10 to reflect new information, submitted by a Grameen Foundation representative, regarding encouraging developments on LAPO since mid-year. To be clear, we stand by the main message of this post, which is not about LAPO’s current situation but about its funders’ and partners’ behavior over the last several years, prior to the public controversy that occurred in late 2009 and early 2010.

We’ve recently released research aiming to identify microfinance institutions (MFIs) with a strong focus on social impact. We have chosen to focus on finding individual MFIs largely because of our concerns about large microfinance-funding charities – specifically, that their due diligence seems focused on financial performance to the exclusion of social impact – i.e., on scale and revenue rather than effects on borrowers’ lives.

A controversy from earlier this year, over a Nigerian MFI called Lift Above Poverty Organization (LAPO), provides a good example of what we’re concerned about. LAPO has been funded and celebrated by many of the big names in microfinance, yet for years there have been many causes for concern about its social (as opposed to financial) performance. From what we’ve seen, it is not clear that these concerns have been on the radar screen of LAPO’s funders and partners.

LAPO’s funders/partners

LAPO’s funders/partners have included:

Controversy and reaction

In August 2009, MicroRate was the first to hint at concerns about LAPO, stating in a press release: “MicroRate notes that the integrity of the information provided to it by LAPO, as well as LAPO’s financial disclosures since the rating, have come into question. As a result, MicroRate’s rating of LAPO is no longer valid.” (MicroRate 2009)

In December 2009, Planet Rating released a “C+” rating report for LAPO that raised substantial concerns about LAPO’s legal licensing, governance, and data integrity, as well as noting an effective annual interest rate in excess of 100%. (Details below.)

In April 2010, The New York Times published an article citing the Planet Rating report on LAPO’s licensing issues and interest rates, while also noting the expiration of the MicroRate rating. Within weeks of this article’s running, both Kiva and MicroPlace had suspended their relationship with LAPO. (See Kiva’s page on LAPO, which discusses the suspension – Kiva’s loans through LAPO appear to have ended in early May – and Microplace’s discussion.) However, the Schwab Foundation award came after the article.

Update 10/19/10: between May 2010 and the present, several encouraging changes at LAPO are reported by a Grameen Foundation representative (via this comment and a followup email on the specifics of dates):

  • May 2010: LAPO “hired Deloitte and Touche to audit its 2010 financials and review the audit that was conducted in 2009.”
  • June 2010: LAPO “received its license from the Nigerian Central Bank and also hired a new Chief Financial Officer, with extensive experience in microfinance management, through the UNDP Africa Management Services Company (AMSCO).”
  • June-September 2010: LAPO “reconstituted its Board of Directors, which now comprises seasoned microfinance, banking and economics professionals from Nigeria and Benin.”
  • October 2010: LAPO “retained the services of consulting firm MicroFinance Transparency (headed by noted expert Chuck Waterfield) to review its interest rates and related policies.”

Below we discuss some details of the concerns, and why we feel they are relevant to our concerns about the due diligence done by LAPO’s funders/partners.

Forced savings and savings without the appropriate license

These issues were a major focus of the Times article, which stated:

    LAPO, considered the leading microfinance institution in Nigeria, engages in a contentious industry practice sometimes referred to as “forced savings.” Under it, the lender keeps a portion of the loan. Proponents argue that it helps the poor learn to save, while critics call it exploitation since borrowers do not get the entire amount up front but pay interest on the full loan.
    LAPO collected these so-called savings from its borrowers without a legal permit to do so, according to a Planet Rating report. “It was known to everybody that they did not have the right license,” Ms. Javoy said.

It appears to us that LAPO has been putting off getting the appropriate license for several years and that its funders have not held it accountable in this regard (though to be clear, it seems possible to us that this failure did not literally constitute breaking the law – we are not sure based on the information we have).

  • The 2005 MicroRate report stated that LAPO was planning to (and should be planning to) become licensed as a Microfinance Bank: “by law, [Community banks] will have to transform into a Microfinance Bank (‘MFB’) by December 2007 … As yet there is no deadline for the transformation of NGOs. However pressure from the central bank is expected and LAPO will have to transform sooner rather than later … The MFI is fully committed to doing so and plans are in place to convert into a private company by the proposed deadline” (page 3). Thus, at this point LAPO appears to have been targeting December 2007 for obtaining its license.

 

  • A letter from the Calvert Foundation concerning its investment in LAPO says the license is hoped for by year-end 2009: “we have been working with the Creditor Taskforce to encourage the transformation of LAPO into a depository institution regulated by the Central Bank as soon as possible. LAPO has received initial approval by the Central Bank for their application to transform into a ‘Microfinance Bank.’ Their goal is to secure the banking license by year-end 2009.“
  • The Planet Rating report, in December 2009, is clear that LAPO still did not have the license at that time, and that it planned to get one in January 2010, a plan that Planet Rating did not find realistic (Planet Rating 2009, Pg 7). Planet Rating stated that “LAPO does not have the appropriate legal structure to … disburse credit or collect savings … Although illegal, this has been so far tolerated by the [Central Bank of Nigeria]” (Pg 7).
  • LAPO still apparently did not have the license as of Kiva’s April 2010 update. This is the most recent discussion we can find of this issue.High interest rates

    The other point emphasized in the Times article is the high rates of interest charged by LAPO, which seem to contradict the stated goals of some of its partners:

      Under outside pressure, LAPO announced in 2009 that it was decreasing its monthly interest rate, Planet Rating noted, but at the same time compulsory savings were quietly raised to 20 percent of the loan from 10 percent. So, the effective interest rate for some clients actually leapt to nearly 126 percent annually from 114 percent, the report said. The average for all LAPO clients was nearly 74 percent in interest and fees, the report found.

    …

    Until recently, Microplace, which is part of eBay, was promoting LAPO to individual investors, even though the Web site says the lenders it features have interest rates between 18 and 60 percent, considerably less than what LAPO customers typically pay.

    …

    At Kiva, which promises on its Web site that it “will not partner with an organization that charges exorbitant interest rates,” the interest rate and fees for LAPO was recently advertised as 57 percent, the average rate from 2007. After The Times called to inquire, Kiva changed it to 83 percent.

    We don’t have much to add on this point. The Planet Rating report specified an effective annual interest rate of 123.9% (Planet Rating 2009, Pg 6).
    We have argued against reading too much into high interest rates, but funders and partners ought to be clear on what these rates are and whether the rates are consistent with their own values. We feel it is very important that anyone funding or partnering with an MFI do the full due diligence required to understand the true effective interest rate, from the beginning of the relationship.

    The two issues raised by the Times article – concern over LAPO’s license and over its interest rates – are both valid issues, and both issues could be easily identified years before the controversy came up.

    Other concerns

    We note other concerns about LAPO’s impact not mentioned in the Times article:

    • Integrity of governance and audits may be compromised by family relationships and other issues. The 2009 Planet Rating report states,
      • “Although the Memorandum of Association states that BOD [Board of Directors] members are to be reelected every year renewed every two years (three years for the chairman), all BOD members have been in the BOD for at least four years” (Pg 4)
      • “One of the [Board of Directors] members is related to the external auditor, creating a risk of lack of transparency. Family relations within the management team create another conflict of interests that have not yet been mitigated by appropriate policies” (Pg 7)
      • “External auditors are not sufficiently independent and do not have enough knowledge on the risks specific to microfinance” (Pg 10)
    • Data is unreliable.
      • “Loan tracking and accounting systems are not integrated and the system is prone to error” (MicroRate 2005, Pg 2)
      • The Planet Rating report stated that information management left room for mistakes and manipulation (Pg 8-9), and that “A sample of six branches by Planet Rating resulted in inconsistencies of up to a 6% difference in the amounts of PAR, arrears and number of clients (as of September 2009)” (Footnote 22). The report warned that “Due to insufficient data reliability, Planet Rating’s opinion on LAPO’s credit risk and credit risk coverage is subject to reserves” (Pg 11).
    • LAPO may lack the tools to assess, and create incentives based on, its social as opposed to financial performance. The Planet Rating report states:

      Group discipline is generally sufficiently ensured. However, for Regular Loans, the evaluation of the borrower’s capacity is not always complete and the actual use of the loan rarely formally monitored. Moreover, LAPO has not defined clear rules for the use of identification papers, which will be necessary to prevent multiple lending as the microfinance market matures and given the multiplication of MFBs … Moreover, the incentive system for Credit Officers mostly relies on their caseload, which creates a risk of excessive disbursements at the expense of portfolio quality. (Page 11)

     

  • High dropout rates. This is the issue that most worried us when we expressed concern about LAPO in December 2009. We cited its 49% dropout rate; as early as 2005, MicroRate stated, “client attrition remains unacceptably high at around 27%” (MicroRate 2005, Pg 5).Bottom line

    We aren’t sure whether/to what extent

    • LAPO’s funders/partners have been largely unaware of/indifferent to the concerns raised above (in some cases, possibly due to prioritizing financial over social returns).
    • LAPO’s funders/partners have been aware and concerned, but have had other, positive information on LAPO’s social impact that they have felt outweighs the concerns.
    • LAPO’s funders/partners have been aware and concerned, but have made a strategic decision to prioritize building sustainable, profitable financial institutions over focusing directly on social impact.

    We feel there is at least some evidence for the first possibility. Two partnerships were suspended in the immediate wake of the Times article, whose major concerns could easily have been identified years ago; and the only public record of due diligence we’re aware of, USAID’s discussion from 2007 (see page 5), discusses only financial/”efficiency” indicators, with no mention of concerns like those listed above.

    The possibility that social performance is essentially being overlooked seems strong and worrisome enough to us that, for the time being, we are more comfortable with the idea of giving directly to MFIs that are clearly focused on their social performance. We are open to changing this view, if and when major microfinance organizations become more open about what factors and concerns they are weighing and how they are conducting their due diligence.

    Sources

 

More on charity ratings and GiveWell’s mission

(updated on: ) | by Holden

Last week we wrote that we are likely going to stop giving zero-to-three-star ratings to charities. Some praised the decision, while others offered suggestions for how to resolve the problems we listed with ratings (we responded to the suggestions in the thread).

While we went into a lot of detail on the problems with ratings, I’ve realized that we didn’t talk enough about the benefits of ratings – and how those benefits don’t fit with GiveWell’s core audience and mission as we’re thinking about it now.

As we mentioned last week, we aren’t getting rid of our charity evaluations, or recommendations, or rankings of recommendations. We’re still going to disclose which charities we like best, and in what order. We’re still going to say what we think of each charity we’ve looked at, whether extensively or briefly.

What we’re getting rid of is the ability to get a quick, quantified rating of any charity we’ve looked at. In some ways that’s a big thing to lose; it’s arguably Charity Navigator‘s greatest strength in attracting broad interest and attention. A lot of the people I’ve talked to about GiveWell aren’t particularly interested in a charity recommendation; they want to know “how good” is the charity their friend is running a marathon for, or the charity that sends them mail. As the Money for Good study puts it, they’re seeking “to validate their donation, not to choose between organizations” (main study, page 40). And sometimes, they also seem to be excited by the possibility of a scandal, i.e., revealing that a particular charity is “bad.”

GiveWell’s mission has never been about serving these people. We seek to drive money to outstanding charities, and in so doing, to change incentives and allow some charities to raise money by doing demonstrably great work (instead of just by telling a great story).

We feel that if all you want from a charity evaluator is to check whether the charity that contacted you is “bad or OK,” you’ve already thrown away most of the opportunity to do as much good as possible with your donation.

For these reasons, our work has always been a poor fit for what a lot of donors want, and there has always been a tension between fulfilling our mission directly (which means focusing obsessively on charities we find promising and ignoring other charities) and doing things that might attract more attention and broaden our audience (such as publishing the star ratings that many want, or “digging up dirt” on big-name charities).

We originally introduced star ratings because we thought they would broaden our reach. The reactions we’ve gotten, particularly regarding how we rate the vast majority of charities that share no substantive information (the toughest aspect of designing an appropriate ratings system), have implied to us that there is actually very little to gain in this way. People who come to us for validation on their existing choice of charity are usually going to see that we don’t find the charity promising. As a result, rather than become interested in the work we do, they’re generally going to be disappointed and even upset.

In many ways, we’re better off making it clear from the beginning that we don’t have what these people are looking for, and focusing exclusively on the donors who do fit what we’re about: coming to their donation decisions with a lack of pre-commitments and an intent to give to the best charity possible.

Why charity ratings don’t work (as of now)

(updated on: ) | by Holden

For a little over a year, GiveWell has assigned zero- to three-star ratings to all charities we’ve examined. We’ve done so in response to constant requests from our fans and followers. We’ve been told that people want easily digested, unambiguous “bottom line” information that can help them make a decision in a hurry and with a clean conscience. We understand this argument. But right now, we feel that the costs of ratings outweigh the benefits, and we’re likely on the brink of getting rid of our ratings.

To be clear, we are not going to stop comparing, evaluating, and recommending charities. As we did for our first couple of years of existence, we will rank and promote a number of recommended charities, while sharing the reasons why we do not recommend other charities. What we are going to stop doing is boiling down our view on a each charity examined into a single quantifiable data point. We’re going to go back to “bottom lines” that are qualified and sometimes difficult to interpret without reading further (for example, instead of “zero stars,” our bottom line will say something more like “Did not pass heuristics to qualify for further investigation”). We know we’ll be sacrificing the kind of simplicity that appeals to many, and we still think it’s worth it.

In trying to provide star ratings, we’ve run into fundamental questions that we don’t have good answers to:

  • Should we rate charities in an “absolute” sense (based on our confidence that they have positive impact) or in a “relative” sense (based on how they compare to other charities working on similar issues)?
  • How should we deal with charities that we feel do excellent work, but have limited or no room for more funding? Should we rate them above or below charities that do less excellent work but have more definite needs? Should our ratings reflect our opinion of organizations’ work or our opinion of whether undecided donors should give to them?
  • The vast majority of charities share no substantive information on their effectiveness, making it impossible to evaluate their effectiveness. Should such charities receive “no rating” (in which case we would rate very few charities, and may provide incentives for charities with low effectiveness to remain opaque) or our lowest rating (which creates considerable offense and confusion among those who feel we have judged their work ineffective)?

Each of these issues involve an ambiguity in what precisely star ratings mean, and we need ways to resolve the ambiguity in a very clear, easily digested, instantly understood way or we lose the benefit we were hoping to gain from handing out ratings in the first place. At this point we cannot construct a system that accomplishes this.

We believe that these issues are unavoidable when assessing charities based on their impact. We believe that nobody else has yet run into these problems because nobody else has yet tried to rate charities based on the case for their impact, i.e., their effects on the people and communities they serve.

Problem 1: are ratings “absolute” or “relative to a cause?”

How does Doctors Without Borders rate? The answer depends partly on whether you’re looking at it as a global health organization or as a disaster relief organization. Compared to other global health organizations, its transparency and documented effectiveness do not seem top-notch (though they are better than average). Compared to other disaster relief organizations (based on our preliminary and subject-to-change impressions), it stands out.

An organization may be top-notch compared to other water organizations, while mediocre in terms of proven health impact. Our view of a charter school organization depends on whether we’re comparing it to other education groups or to U.S. equality of opportunity organizations of all kinds. The more one tries to accommodate wishes like fighting a specific disease or attacking a problem in a specific way – i.e., the more one explores and subdivides different causes – the more these difficult questions come up.

We have been rating each organization “relative to” the cause in which it seems to fit most intuitively. However, this is confusing for donors who don’t have strong cause-based preferences and take a broad view of charity as “helping people in general.” (Usually these are the donors who are a particularly good fit for what we provide.) Alternately, we could rate each organization using an “absolute” scale (taking the cause into account), but if we did this we’d rank even relatively mediocre international aid charities above the outstanding Nurse-Family Partnership, and that would create considerable confusion among people who didn’t agree with our (highly debatable) view on international vs. domestic aid.

In the end we don’t feel comfortable rating Nurse-Family Partnership higher than Small Enterprise Foundation … or lower … or the same. They’re too different; your decision on which to give to is going to come down to judgment calls and personal values.

It is possible for ratings systems to deal effectively with “apples and oranges” comparisons. Consumer websites (e.g., Amazon) provide ratings for products in many different categories; consumers generally seem to understand that the ratings capture something like “how the product performs relative to expectations,” and expect to supplement the ratings with their own thoughts about what sort of product and what features they want. However, in this domain I feel that consumers generally have a good feel for what different product categories and features consist of (for example, I know what to expect from a laser vs. inkjet printer, and don’t assume that this issue is captured in the rating). In the charity world, there is often just as little to go on regarding “what can be expected from an education charity?” as there is regarding “which education charity is best of the bunch?” So there is ambiguity regarding the extent to which a rating includes our view of the charity’s general cause.

While this problem isn’t a fatal one for charity ratings, it brings some complexity and confusion that is compounded by the issues below.

Problem 2: do ratings incorporate whether a group has room for more funding?

We’ve argued before that the issue of room for more funding is drastically underappreciated and under-discussed, and it creates major challenges for a ratings system.

The question is how to rate an organization such as Aravind Eye Care System, AMK or (arguably) Nurse-Family Partnership – an organization that we largely think is doing excellent work, but has limited room for more funding. On one hand, we need donors to know that their money may be more needed/productive elsewhere; giving a top-notch organization a top-notch rating does not communicate this. On the other hand, if we were to lower Nurse-Family Partnership’s rating, that would imply to many that we do not have as high an opinion of their work, and may even result in reduced support from existing donors, something we definitely don’t want to see happening.

Then there are organizations which we do not investigate, even though they are promising and pass our initial heuristics, because it comes out early in the process that they have no room for more funding. We therefore have no view of these organizations’ work, one way or the other; we simply know that they are not a good fit for the donors using our recommendations.

The ambiguity here is regarding whether ratings represent our view of an organization’s work or our view of it as an opportunity for non-preexisting donors.

Problem 3: how should we rate the vast majority of charities that share no substantive information?

If a charity doesn’t collect, and share, substantive information on its effectiveness, there is no way of gauging its effectiveness. From what we’ve seen, the vast majority of charities do not both collect and share substantive information on their effectiveness. This gives us two unattractive options:

1. Give ratings only to charities that share enough information to make it possible to gauge their impact. If we did this, we would have a tiny set of rated charities, with all the rest (including some of the largest and least transparent charities such as UNICEF) marked as “Not rated.” Our lowest-rated charities would in fact be among the most transparent and accountable charities; we would effectively be punishing charities for sharing more information; people who wanted to know our view of UNICEF would wrongly conclude that we had none.

2. Give our lowest rating to any charity that shares no substantive information. This is the approach we have taken. This results in the vast majority of our ratings being “zero stars,” something that makes many donors and charities uncomfortable and leads to widespread confusion. Many people think that a “zero star” rating indicates that we have determined that a group is doing bad work, when in fact we simply don’t have the information to determine its effectiveness one way or the other. We have tried to reduce confusion by modeling our ratings on the zero-to-three-star Michelin ratings (where the default is zero stars and even a single star is a positive mark of distinction) rather than on the more common one-to-five-star system (where one star is a mark of shame), but confusion persists.

Bottom line

All of the above issues involve ambiguity in how our ratings should be read. Any of them might be resolvable, on its own, with some creative presentation. However, we have not been able to come up with any system for addressing all of these issues that remains simple and easily digestible, as ratings should be. So we prefer, at least for the time being, to sacrifice simplicity and digestibility in favor of clear communication. Rather than giving out star ratings, we will provide more complex and ambiguous bottom lines that link to our full reviews.

We understand that this is a debatable decision. We wish to identify outstanding charities and drive money to them; we wish to have a reputation for reliability and integrity among thoughtful donors; the goal of giving large numbers of people a bottom-line rating on the charity of their choice is less important to us. We know that other organizations may make the tradeoff differently and don’t feel it is wrong to do so.

Note that we haven’t finalized this decision. We welcome feedback on how to resolve the tensions above with a simple, easily understood ratings system.