We often use academic research to inform our work, but we try to do so with great caution, rather than simply taking reported results at face value. We believe that if you trust academic research just because it is peer-reviewed, published, and/or reputable, this is a mistake.

A good example of why we’re concerned comes from the recent back-and-forth between David Roodman and Mark Pitt, which continues a debate begun in 1999 over what used to be considered the single best study on the social impact of microfinance.

It appears that the leading interpretation of this study swung wildly back and forth over the course of a decade, based not on major reinterpretations but on arguments over technical details, while those questioning the study were unable to view the full data and calculations of the original. We feel that this illustrates problems with taking academic research at face value and supports many of the principles we use in our approach to using academic research. Details follow.

1998-2005 studies by Khandker and Pitt

According to a 2005 white paper published by the Grameen Foundation (PDF), a 1998 book and accompanying paper released by Shahidur Khandker and Mark Pitt “were influential because they were the first serious attempt to use statistical methods to generate a truly accurate assessment of the impact of microfinance.”

Jonathan Morduch challenged these findings shortly after their publication, but a 2005 followup by Khandker appeared to answer the challenge and claim that microlending had a very strong social impact:

each additional 100 taka of credit to women increased total annual household expenditures by more than 20 taka … microfinance accounted for 40 percent of the entire reduction of moderate poverty in rural Bangladesh.

As far as we can tell, this result stood for about four years as among the best available evidence that microlending helped bring people out of poverty. Our mid-2008 review of the evidence stated,

These studies rely heavily on statistical extrapolation about who would likely have participated in programs, and they are far from the strength and rigor of the Karlan and Zinman (2007) study listed above, but they provide somewhat encouraging support for the idea that the program studied had a widespread positive effect.

2009 response by Roodman and Morduch

A 2009 paper by David Roodman and Jonathan Morduch argued that

• The Khandker and Pitt studies were seriously flawed in their attempts to attribute impact. The reduction in poverty they observed could have been an artifact of wealth driving borrowing, rather than the other way around.
• The Khandker and Pitt studies could not be replicated: the full data and calculations they had used were not public, and Roodman and Morduch’s best attempts at a replication did not produce a remotely similar conclusion (they demonstrated no positive social impact of microlending, even a slight negative one).

This paper stood for the next two years as a prominent refutation of the Khandker and Pitt studies. Pitt writes that the work of Roodman and Morduch has become “well-known in academic circles” and “seems to have had a broad impact.” It appeared in a “new volume of the widely respected Handbook of Development Economics” as well as in congressional testimony.

2011 back-and-forth

Earlier this year,

• Mark Pitt published a response arguing that Roodman and Morduch’s failure to replicate his study was due to Roodman and Morduch’s errors.
• David Roodman replied, conceding an error in his original replication but defending his claim that the original study (by Khandker and Pitt) was not a valid demonstration of the impact of microlending.
• Mark Pitt responded again and argued that the study was a valid demonstration.
• David Roodman defended his statement that it was not and added, “this is the first time someone other than [Mark Pitt] has been able to run and scrutinize the headline regression in the much-discussed paper … If you anchor Pitt and Khandker’s regression properly in the half-acre rule … the bottom-line impact finding goes away.”
• We had hoped to see a further response from Mark Pitt before discussing this matter, but Roodman also wrote that Mark Pitt is now traveling and that “this could be the last chapter in the saga for a while.”

Bottom line: as far as we can tell, we still have one researcher claiming that the original study strongly demonstrates a positive social impact of microfinance; another researcher claiming it demonstrates no such thing; and no end in sight, 13 years after the publication of the original study.

Disagreements among researchers are common, but this one is particularly worrisome for a few reasons.

• Conflicting interpretations of the study have each stood for several years at a time. The original study stood as the leading evidence about microlending’s social impact between 2005-2009; the challenge by Roodman and Morduch was highly prominent, and apparently not commented on at all by the original authors, between 2009-2011.

• Disagreements have been technical, many concerning details that few understand and that still don’t seem resolved. David Roodman states that “the omission of the dummy for a household’s target status” is responsible for his estimated effect of microlending coming out negative instead of positive. Numerous other errors on both sides are alleged, and the remaining disagreements over causal inference are certainly beyond what I can easily follow (if a reader can explain them in clear terms I encourage doing so in the comments).
• Resolution has been hampered by the fact that Roodman and Morduch could only guess at the calculations Pitt and Khandker performed. This is the biggest concern to me. Roodman writes that he was never able to obtain the original data set used in the paper; that the data set he did receive (upon request) was (in his view) confusingly labeled; and even that one of the original authors “fought our efforts to obtain the later round of survey data from the World Bank.” As a result, his attempt at replication was a “scientific whodunit,” and his April 2011 update represents “the first time someone other than [the original author] has been able to run and scrutinize the headline regression in the much-discussed paper.”

  If I weren’t already somewhat familiar with this field, I would be shocked that it’s even possible to have a study accepted to any journal (let alone a prestigious one) without sharing the full details of the data and calculations, and having the calculations replicated and checked. But in fact, disclosure of data - and replication/checking of calculations - appears to be the exception, not the rule, and is certainly not a standard part of the publication/peer review process.

Bottom line - the leading interpretation of a reputable and important study swung wildly back and forth over the course of a decade, based not on revolutionary reinterpretations but on quibbles over technical details, while no one was able to view the full data and calculations of the original. For anyone assuming that a prestigious journal’s review process - or even a paper’s reputation - is a sufficient stamp of reliability on a paper, this is a wake-up call.

• Never put too much weight on a single study. If nothing else, the issue of publication bias makes this an important guideline. (On this note, note that the 2009 Roodman and Morduch paper was rejected for publication; its sole peer-reviewer was an author of the original paper that Roodman and Morduch were questioning.)
• Strive to understand the details of a study before counting it as evidence. Many “headline claims” in studies rely on heavy doses of assumption and extrapolation. This is more true for some studies than for others.
• If a study’s assumptions, extrapolations and calculations are too complex to be easily understood, this is a strike against the study. Complexity leaves more room for errors and judgment calls, and means it’s less likely that meaningful critiques have had the chance to emerge. Note that before the 2009 response to the study discussed here was ever published, GiveWell took it with a grain of salt due to its complexity (see quote above). Randomized controlled trials tend to be relatively easy to understand; this is a point in their favor.
• If a study does not disclose the full details of its data and calculations, this is another strike against it - and this phenomenon is more common than one might think.
• Context is key. We often see charities or their supporters citing a single study as “proof” of a strong statement (about, for example, the effectiveness of a program). We try not to do this - we generally create broad overviews of the evidence on a given topic and source our statements to these.

While a basic fact can be researched, verified and cited quickly, interpreting an impact study with appropriate care takes - in our view - concentrated time and effort and plenty of judgment calls. This is part of why we’re less optimistic than many about the potential for charity research based on (a) crowdsourcing; (b) objective formulas. Instead, our strategy revolves around transparency and external review.

Updated 10/19/10 to reflect new information, submitted by a Grameen Foundation representative, regarding encouraging developments on LAPO since mid-year. To be clear, we stand by the main message of this post, which is not about LAPO’s current situation but about its funders’ and partners’ behavior over the last several years, prior to the public controversy that occurred in late 2009 and early 2010.

We’ve recently released research aiming to identify microfinance institutions (MFIs) with a strong focus on social impact. We have chosen to focus on finding individual MFIs largely because of our concerns about large microfinance-funding charities - specifically, that their due diligence seems focused on financial performance to the exclusion of social impact - i.e., on scale and revenue rather than effects on borrowers’ lives.

A controversy from earlier this year, over a Nigerian MFI called Lift Above Poverty Organization (LAPO), provides a good example of what we’re concerned about. LAPO has been funded and celebrated by many of the big names in microfinance, yet for years there have been many causes for concern about its social (as opposed to financial) performance. From what we’ve seen, it is not clear that these concerns have been on the radar screen of LAPO’s funders and partners.

LAPO’s funders/partners

LAPO’s funders/partners have included:

• Grameen Foundation, which has partnered with LAPO since 2003 and gave it its 2006 Excellence in Microfinance Award.
• UNDP, Ford Foundation, and USAID, all of which have provided grant funding (Planet Rating 2009, Pg 2).
• The Oxfam Novib fund, a member of Oxfam International (see LAPO’s Partners page).
• Deutsche Bank (according to the New York Times)
• Two “giving marketplaces”: Kiva and Microplace (again see LAPO’s Partners page).
• LAPO’s founder also received the Schwab Foundation’s Regional Social Entrepreneur of the Year award.

Controversy and reaction

In August 2009, MicroRate was the first to hint at concerns about LAPO, stating in a press release: “MicroRate notes that the integrity of the information provided to it by LAPO, as well as LAPO’s financial disclosures since the rating, have come into question. As a result, MicroRate’s rating of LAPO is no longer valid.” (MicroRate 2009)

In December 2009, Planet Rating released a “C+” rating report for LAPO that raised substantial concerns about LAPO’s legal licensing, governance, and data integrity, as well as noting an effective annual interest rate in excess of 100%. (Details below.)

In April 2010, The New York Times published an article citing the Planet Rating report on LAPO’s licensing issues and interest rates, while also noting the expiration of the MicroRate rating. Within weeks of this article’s running, both Kiva and MicroPlace had suspended their relationship with LAPO. (See Kiva’s page on LAPO, which discusses the suspension - Kiva’s loans through LAPO appear to have ended in early May - and Microplace’s discussion.) However, the Schwab Foundation award came after the article.

Update 10/19/10: between May 2010 and the present, several encouraging changes at LAPO are reported by a Grameen Foundation representative (via this comment and a followup email on the specifics of dates):

• May 2010: LAPO “hired Deloitte and Touche to audit its 2010 financials and review the audit that was conducted in 2009.”
• June 2010: LAPO “received its license from the Nigerian Central Bank and also hired a new Chief Financial Officer, with extensive experience in microfinance management, through the UNDP Africa Management Services Company (AMSCO).”
• June-September 2010: LAPO “reconstituted its Board of Directors, which now comprises seasoned microfinance, banking and economics professionals from Nigeria and Benin.”
• October 2010: LAPO “retained the services of consulting firm MicroFinance Transparency (headed by noted expert Chuck Waterfield) to review its interest rates and related policies.”

Below we discuss some details of the concerns, and why we feel they are relevant to our concerns about the due diligence done by LAPO’s funders/partners.

Forced savings and savings without the appropriate license

These issues were a major focus of the Times article, which stated:

LAPO, considered the leading microfinance institution in Nigeria, engages in a contentious industry practice sometimes referred to as “forced savings.” Under it, the lender keeps a portion of the loan. Proponents argue that it helps the poor learn to save, while critics call it exploitation since borrowers do not get the entire amount up front but pay interest on the full loan.

LAPO collected these so-called savings from its borrowers without a legal permit to do so, according to a Planet Rating report. “It was known to everybody that they did not have the right license,” Ms. Javoy said.

It appears to us that LAPO has been putting off getting the appropriate license for several years and that its funders have not held it accountable in this regard (though to be clear, it seems possible to us that this failure did not literally constitute breaking the law - we are not sure based on the information we have).

• The 2005 MicroRate report stated that LAPO was planning to (and should be planning to) become licensed as a Microfinance Bank: “by law, [Community banks] will have to transform into a Microfinance Bank (’MFB’) by December 2007 … As yet there is no deadline for the transformation of NGOs. However pressure from the central bank is expected and LAPO will have to transform sooner rather than later … The MFI is fully committed to doing so and plans are in place to convert into a private company by the proposed deadline” (page 3). Thus, at this point LAPO appears to have been targeting December 2007 for obtaining its license.

• A letter from the Calvert Foundation concerning its investment in LAPO says the license is hoped for by year-end 2009: “we have been working with the Creditor Taskforce to encourage the transformation of LAPO into a depository institution regulated by the Central Bank as soon as possible. LAPO has received initial approval by the Central Bank for their application to transform into a ‘Microfinance Bank.’ Their goal is to secure the banking license by year-end 2009.”
• The Planet Rating report, in December 2009, is clear that LAPO still did not have the license at that time, and that it planned to get one in January 2010, a plan that Planet Rating did not find realistic (Planet Rating 2009, Pg 7). Planet Rating stated that “LAPO does not have the appropriate legal structure to … disburse credit or collect savings … Although illegal, this has been so far tolerated by the [Central Bank of Nigeria]” (Pg 7).
• LAPO still apparently did not have the license as of Kiva’s April 2010 update. This is the most recent discussion we can find of this issue.

High interest rates

The other point emphasized in the Times article is the high rates of interest charged by LAPO, which seem to contradict the stated goals of some of its partners:

Under outside pressure, LAPO announced in 2009 that it was decreasing its monthly interest rate, Planet Rating noted, but at the same time compulsory savings were quietly raised to 20 percent of the loan from 10 percent. So, the effective interest rate for some clients actually leapt to nearly 126 percent annually from 114 percent, the report said. The average for all LAPO clients was nearly 74 percent in interest and fees, the report found.

…

Until recently, Microplace, which is part of eBay, was promoting LAPO to individual investors, even though the Web site says the lenders it features have interest rates between 18 and 60 percent, considerably less than what LAPO customers typically pay.

…

At Kiva, which promises on its Web site that it “will not partner with an organization that charges exorbitant interest rates,” the interest rate and fees for LAPO was recently advertised as 57 percent, the average rate from 2007. After The Times called to inquire, Kiva changed it to 83 percent.

We don’t have much to add on this point. The Planet Rating report specified an effective annual interest rate of 123.9% (Planet Rating 2009, Pg 6).
We have argued against reading too much into high interest rates, but funders and partners ought to be clear on what these rates are and whether the rates are consistent with their own values. We feel it is very important that anyone funding or partnering with an MFI do the full due diligence required to understand the true effective interest rate, from the beginning of the relationship.

The two issues raised by the Times article - concern over LAPO’s license and over its interest rates - are both valid issues, and both issues could be easily identified years before the controversy came up.

Other concerns

We note other concerns about LAPO’s impact not mentioned in the Times article:

• Integrity of governance and audits may be compromised by family relationships and other issues. The 2009 Planet Rating report states,
  • “Although the Memorandum of Association states that BOD [Board of Directors] members are to be reelected every year renewed every two years (three years for the chairman), all BOD members have been in the BOD for at least four years” (Pg 4)
  • “One of the [Board of Directors] members is related to the external auditor, creating a risk of lack of transparency. Family relations within the management team create another conflict of interests that have not yet been mitigated by appropriate policies” (Pg 7)
  • “External auditors are not sufficiently independent and do not have enough knowledge on the risks specific to microfinance” (Pg 10)
• Data is unreliable.
  • “Loan tracking and accounting systems are not integrated and the system is prone to error” (MicroRate 2005, Pg 2)
  • The Planet Rating report stated that information management left room for mistakes and manipulation (Pg 8-9), and that “A sample of six branches by Planet Rating resulted in inconsistencies of up to a 6% difference in the amounts of PAR, arrears and number of clients (as of September 2009)” (Footnote 22). The report warned that “Due to insufficient data reliability, Planet Rating’s opinion on LAPO’s credit risk and credit risk coverage is subject to reserves” (Pg 11).
• LAPO may lack the tools to assess, and create incentives based on, its social as opposed to financial performance. The Planet Rating report states:
  

  Group discipline is generally sufficiently ensured. However, for Regular Loans, the evaluation of the borrower’s capacity is not always complete and the actual use of the loan rarely formally monitored. Moreover, LAPO has not defined clear rules for the use of identification papers, which will be necessary to prevent multiple lending as the microfinance market matures and given the multiplication of MFBs … Moreover, the incentive system for Credit Officers mostly relies on their caseload, which creates a risk of excessive disbursements at the expense of portfolio quality. (Page 11)

• High dropout rates. This is the issue that most worried us when we expressed concern about LAPO in December 2009. We cited its 49% dropout rate; as early as 2005, MicroRate stated, “client attrition remains unacceptably high at around 27%” (MicroRate 2005, Pg 5).

Bottom line

We aren’t sure whether/to what extent

• LAPO’s funders/partners have been largely unaware of/indifferent to the concerns raised above (in some cases, possibly due to prioritizing financial over social returns).
• LAPO’s funders/partners have been aware and concerned, but have had other, positive information on LAPO’s social impact that they have felt outweighs the concerns.
• LAPO’s funders/partners have been aware and concerned, but have made a strategic decision to prioritize building sustainable, profitable financial institutions over focusing directly on social impact.

We feel there is at least some evidence for the first possibility. Two partnerships were suspended in the immediate wake of the Times article, whose major concerns could easily have been identified years ago; and the only public record of due diligence we’re aware of, USAID’s discussion from 2007 (see page 5), discusses only financial/”efficiency” indicators, with no mention of concerns like those listed above.

The possibility that social performance is essentially being overlooked seems strong and worrisome enough to us that, for the time being, we are more comfortable with the idea of giving directly to MFIs that are clearly focused on their social performance. We are open to changing this view, if and when major microfinance organizations become more open about what factors and concerns they are weighing and how they are conducting their due diligence.

Sources

• MicroRate report on LAPO - December 2005
• USAID Nigeria Audit of Microfinance Activities - November 2007
• MicroRate press release on LAPO - August 2009
• Calvert Letter regarding LAPO - November 2009
• Planet Rating report on LAPO - December 2009 (this report is not publicly available; we purchased one license and cannot post the document)

Over the past few months, we’ve been continuing our search for outstanding microfinance organizations (in addition to the one we’ve already identified). Below are the results.

Overview of our process and key questions

In brief, our take on microfinance is that offering credit and other financial services is likely an effective way to improve people’s lives in the developing world. At the same time, providing credit carries with it the risk of causing harm to clients. Donors therefore should carefully choose the microfinance institutions (MFIs) they choose to support, focusing in particular on an MFI’s demonstrated focus on (a) effectively providing credit while (b) assessing clients’ well-being and avoiding causing harm.

When we contact an MFI, we ask them a set of questions to evaluate them on these criteria. In particular, we assess:

• Focus on social impact. The primary issue we ask MFIs about is whether and to what degree they track clients who drop out of the program (i.e., complete a loan cycle and choose not to take out subsequent loans). As we’ve written before, high dropout rates may be a sign that clients are having bad experiences and/or finding that the benefits of loans don’t compensate for the (often high) interest rates. We try to determine an organization’s degree of focus on dropouts by asking about (a) the dropout rate and how it’s calculated, (b) how the dropout rate is used in internal evaluation (e.g., is it used to inform employee compensation? branch-level performance?), and (c) whether the organization performs in-depth surveys that focus on the reasons why borrowers drop out. We believe that MFIs who thoroughly track those who choose to leave the program are most likely to identify and address problems clients have with the MFI’s services.

  We don’t only ask about the dropout rate. Some MFIs take other measures to determine whether they’re causing clients problems - for example, MFIs may attempt to ascertain whether clients are borrowing from multiple MFIs (e.g., taking on too much debt), or they may conduct regular surveys of clients’ satisfaction.

• Interest rates. Borrowers at MFIs pay interest rates that most of us would consider unthinkably high. “Normal” rates tend to be in the 40-100% range (that’s the annualized equivalent in the terms used in the U.S.); and we’ve seen rates as high as 150% annualized. Because the way MFIs report interest rates varies — some require clients to save to effectively create collateral in the event they default; others add fees on the front of loans which may not be included in the headline rates — we’ve asked all the MFIs we’ve considered to provide us with enough detail to calculate their APR and EIR so that we can provide donors with information about the rates borrowers are paying at each institution.
• Room for more funding. As with any organization we look at, we assess whether the institution can effectively utilize additional funds and how those funds will be used. In many cases, we’ve found MFIs that can support continuing operations with revenues and don’t require donations to maintain or expand their operations.
• Repayment rate and clients’ standard of living. We seek evidence that clients are repaying their loans consistently and that MFIs are generally serving people who have low incomes. Most of the MFIs we’ve contacted can provide reasonable evidence that the people they’re serving are poor and that those who borrow generally repay their loans (note, however, that one of our major criteria for contacting MFIs was that they report collecting evidence on clients’ standards of living, so it isn’t necessarily the case that most MFIs in general meet this criterion).

Results

We chose to contact MFIs listed on Mix Market that we thought would have a good chance of answering our questions well. For more detail on how we chose MFIs to contact and which MFIs we contacted and spoke to, see the page explaining our process for finding microfinance charities. In all, we’ve contacted 43 MFIs; we were able to speak with 18, and 11 provided us with enough information to complete an in-depth review.

The first table below shows each MFI’s answers to our key questions. The asterisks represent the quality of the information we received: *** = high quality information; ** = medium quality; * = low quality. The table also links to our review pages for each MFI in cases where the review is complete and we have permission to publish it. We haven’t yet completed our review of AMK.

Notes:

• PAR>30 and write-off ratios are not given quality ratings because they are all taken directly from Mix Market, and thus we are not aware of any variation in quality. They are for the most recent year for which data is available (2008 or 2009). They do not describe the current portfolio of any MFI.
• For more information on what we mean by a “collection rate,” see our blog post, “More on the microfinance repayment rate.”
• For more information on different methods for calculating interest rates, see our post, “Microfinance interest rates.”
• For more information on the standard of living information we used for each MFI, see this excel file.
• We didn’t ask ID-Ghana for information on their interest rates. At the time we reviewed them (late-2009), interest rates were not a key step in our process.

Based on this information, there are certain MFIs that we think stand out for the purposes of an individual donor seeking a group with a strong focus on social impact.

Note: AMK appears strong on all factors we investigated (to the extent we investigated them), but informed us that it was recently sold to an equity fund, and it is therefore unclear to us what role donations will play in AMK’s operations in the future. Note that AMK is listed as one of Kiva’s largest partners, and likely “effectively” receives donations through that vehicle (since it charges substantial interest while not paying interest in Kiva loans).

It seems like no one is sure why Unitus is closing its doors. That said - what can we learn from this situation if, as stated, Unitus is closing down because it has accomplished its mission and no longer needs to exist?

“We have always thought of Unitus as a project, and that when we completed the project, we would have the integrity to say we were done,” says Joseph Grenny, one of the seven founders, and the chair of Unitus’s board. (From the Chronicle of Philanthropy’s report on Unitus)

From a donor perspective, what this quote is describing is the issue of room for more funding, which we’ve discussed at length (see our page and blog post series on this issue). No matter how successful a program is, there are limits to how much it can be productively expanded.

In December, we argued that room for more funding is a key question few others are asking charities. The Unitus case – if they did in fact shut down because they “completed the project” – lends support to our argument.

• Last year, Philanthropedia polled 130 professionals in the sector; Unitus came up as a top recommendation.
• The Omidyar Network has granted $11.7 million to Unitus since 2005. (See the Chronicle story.)

We can’t find evidence that Unitus itself gave an indication that its use for more funds was limited. According to the Puget Sound Business Journal, Unitus was “interviewing potential candidates for its vacant top fundraising post as recently as a month ago.”

Donors shouldn’t rely on a charity to tell them when it’s running out of room for more funding. We believe that the issue of “room for more funding” is one of the most under-recognized issues in the field of charity evaluation, especially for individual donors.

Unitus today released a rather sudden announcement that it will be releasing its staff and “shift[ing] its resources and activities to areas of maximum socio-economic impact for underserved people throughout the world that have yet to attain either scale or commercial viability.”

I agree with Sean Stannard-Stockton that while the announcement presents the closing as a simple strategic change in direction (due to success, rather than failure), its suddenness is odd and raises many questions.

We have always found Unitus to be an opaque organization. It is one of several large U.S. microfinance organizations whose value-added is unclear.

We urge Unitus to provide more information than is in its press release.

If the “change of direction” is in response to failures and/or disappointments, we urge Unitus to be clear on this point and to share as much information as possible about what has happened so that others can learn from its experience.

If the “change of direction” is, as the press release implies, a big-picture strategic change brought about by less need for its activities, we have the following questions:

• What, specifically, will Unitus be focusing on from here?
  
• Why does its new focus require a complete change in personnel?

• Why has the announcement been made so suddenly? Recent newsletters and press releases seem to imply that there are substantial unexploited opportunities for the work Unitus is doing; did these give the wrong impression? What has changed?

Coming on the heels of the LAPO controversy and our recent post about USAID’s “failure” to target the poorest, we see this event as yet another reason to be wary of the appealing stories associated with microfinance.

USAID’s most recent report on microfinance and microenterprise development tells an interesting story and, in my view, shows just how widely microfinance has been (and continues to be) misunderstood. While many advocate that microfinance institutions focus on people under the global “extreme poverty line”, USAID’s report implies that actually doing so is rare and even unrealistic.

Background: the myth of targeting the poorest

The international “extreme poverty line” is around the equivalent of US$1.25 per day, and around 1.4 billion people worldwide (and over half of those in sub-Saharan Africa) are estimated to live below this line (see the discussion in our international aid report).

Many seem to believe that people in this category are appropriate - even ideal - as clients. For example, see Opportunity International and Grameen Foundation stressing the need to reach the “poorest” and “most vulnerable.” Both Accion and CGAP cite the entire 3-billion-strong set of people under the US$2/day line as potential microfinance clients (upwards of 50% of this set falls below the “extreme poverty line”).

U.S. official aid seems to have taken this idea particularly far. For the past several years, USAID has been required by law to target the “very poor,” defined partly with reference to this “extreme poverty line”:

Both the Microenterprise for Self Reliance Act of 2000 (henceforth, the 2000 Act) and the MRAA mandate that at least half of all USAID funding for microenterprise development directly benefit the very poor. The 2000 Act initially defined the “very poor” as the bottom [poorest] half of those living below each country’s national poverty line … Subsequent amendments to the 2000 Act mandated a second, much more ambitious approach … the amended law created a second definition of the “very poor” — those living on less than the equivalent of $1 per day, calculated using purchasing power parity (PPP) exchange rates. The law made clear that, for any given country, the applicable definition of the very poor would be the more inclusive one.

(Note that “$1/day” may be a reference to the $1.25/day “extreme poverty line” discussed here - see note 6.)

Investigating actual poverty levels of clients

To its credit, USAID put significant effort into tracking whether it was actually meeting this goal, developing poverty assessment tools for assessing clients’ poverty levels and requiring certain grantees to use them. The results:

Among the eight microfinance institutions that applied and reported on the Poverty Assessment Tools, the average share of Funds Benefiting the Very Poor (FVP) is estimated at 28.5 percent, up from 16.3 percent in FY 2007. … For the 14 enterprise development programs that applied and reported on the Poverty Assessment Tools, average FVP is estimated at 26.0 percent, up from 20.5 percent in FY 2007 …

USAID did not come close to its target of 50% “extremely poor” clients. Furthermore, it concluded that continuing to push for this target would be unwise:

As matters stand, USAID sees no promising options for meeting the FVP target. It cannot do so by reallocating funds among its existing partners, because with the exception of one small program, none had more than 50 percent “very poor” clients. It cannot do so by shifting funds to established microenterprise organizations that are not already receiving USAID funding, because few if any such organizations are voluntarily applying the USAID-certified poverty assessment tools, and no such organization has offered solid evidence that it has more than 50 percent “very poor” clients …

Misguided target?

USAID does not conclude that microfinance/microenterprise projects should be de-emphasized (it observes that “the great majority of clients … are very poor, at least in commonly used terms”). Instead, it concludes that the idea of serving the poorest was unrealistic/inappropriate in the first place.

the overall pattern of results lend further weight to the point that USAID raised in last year’s Annual Report - that current law imposes too low a threshold for being “very poor.” This very narrow definition makes it impossible for USAID to allocate its microfinance and microenterprise funding so as to reach the legislative target of directing 50 percent of the benefits of microenterprise funding to the “very poor,” without undermining other goals emphasized in the same legislation, such as sustainability and support for broad-based economic growth.

Unfortunately, this definition of being “very poor” was adopted without any evidence that a 50 percent FVP target based on this definition could be reached. Two years of results using the poverty assessment tools strongly suggest that the target cannot be reached without inflicting undesirable side effects on sustainability and economic development. In short, USAID sees no realistic prospect of reaching the target contained in the law, and urges prompt and serious consideration of changes in the law. (Bold mine; italics in original)

I’m inclined to agree with USAID’s conclusion. I agree that people with incomes well above the “extreme poverty line” can still be very poor, certainly poor enough that I’m interested in donating to help them. So my point is not that microfinance is being carried out inappropriately, or is failing to reach the very poor.

Rather, I’m noting yet another way in which microfinance seems to have been badly misunderstood by its biggest funders and proponents. USAID, and by implication its grantees, seem to have thought that they were serving the world’s poorest - to the point of legislating it - without any data, and wrongly. It’s another debunked myth, and another sign that the funding and the stories have gotten ahead of the facts.

Back in December, we expressed concerns about LAPO, one of Kiva’s largest microfinance partners. Last month, the New York Times ran an article implying criticism of LAPO. Now, Kiva has suspended its partnership with LAPO. A couple of questions this raises:

Which of the several objections to LAPO have led to the suspension?

Several concerns have been raised about LAPO:

• The New York Times article focuses largely on the high interest rates and forced savings (which cause the interest rates to be higher). We have argued that focusing on interest rates in isolation is a mistake.
• The Times article also raised concerns about LAPO’s transparency/accountability, including the question of whether it is collecting savings without the right legal license and the question of whether it is misleadingly presenting its interest rates to the outside world.
• To us, the most compelling concern about LAPO is its 49% dropout rate.

How many more of Kiva’s partners might have similar problems, and what is Kiva doing to address this concern?

As the case of LAPO shows, the interest rates reported to Kiva may not fully capture what clients are paying. More broadly, it appears to me that Kiva’s due diligence on partners is intended to assess the risk of money being lost and/or going to illegitimate organizations, but not the more difficult-to-assess risk encountered with LAPO: that a partner’s activities may be leading to overindebted, misinformed, and/or dissatisfied clients, and thus result in financial success but not positive social impact. The stories posted on Kiva’s website do not, in my view, help to address these risks either.

When the New York Times discovers a problem, makes it very public and causes change, this is a good thing. But I don’t think we can expect this mechanism to reliably address the broader issue. Kiva has had enormous success getting people excited about microfinance and getting capital to MFIs; I would like to see more work go into making sure those MFIs are benefiting the people they serve.

I’m very interested in the recent debate over microfinance interest rates (see our response to the NYT article, as well as Te-Ping Chen’s comments at Change.org).

It seems that realizing how high interest rates can be has been a wake-up call to many that microfinance can easily be doing damage as well as good. If someone is paying 150% interest a year, all it takes is some accounting errors for them to end up losing money and getting stuck in debt rather than helping themselves as intended.

But isn’t this concern equally valid for lower interest rates? The same people complaining about excessive interest rates imply that rates 10-30% above the cost of raising capital can be considered reasonable. 10-30% above the cost of raising capital is still significant - it’s more than most Americans pay on their credit card debt - and small losses could be especially risky and damaging to the very poor.

In my view, there is no substitute for asking tough questions about social impact and no excuse for donating to a microfinance charity that can’t answer them.

Recent coverage of microfinance has had a sharp focus on interest rates, implying some line between “reasonable” interest (associated with “social investment”) and “excessive” interest (associated with “loan sharking”).

In Nicaragua, President Daniel Ortega, outraged that interest rates there were hovering around 35 percent in 2008, announced that he would back a microfinance institution that would charge 8 to 10 percent, using Venezuelan money …

Damian von Stauffenberg, who founded an independent rating agency called Microrate, said that local conditions had to be taken into account, but that any firm charging 20 to 30 percent above the market was “unconscionable” and that profit rates above 30 percent should be considered high.

Mr. Yunus says interest rates should be 10 to 15 percent above the cost of raising the money, with anything beyond a “red zone” of loan sharking. “We need to draw a line between genuine and abuse,” he said. “You will never see the situation of poor people if you look at it through the glasses of profit-making.”

It seems very important that interest rates be transparent, i.e., clearly communicated to and understood by clients. It also is clearly important that there be no coercion, i.e., that clients not be pressured to take loans they don’t want to take. More debatable, but something that we support strongly, are additional measures to assess and improve the client experience, including monitoring overindebtedness, examining dropout rates, etc.

But if/when such things are in place, it is unclear to me on what grounds anyone can complain about interest rates being “too high.” If the terms of loans are clearly communicated, then I see no explanation for why clients would take out loans - unless they feel they have no better alternatives.

What objection can be raised to a 100% interest rate, if the next-best alternative is a 500% interest rate (as I have been told some informal moneylenders charge)? What objection can be raised to a 500% interest rate, if there is no other way for people to get credit? When a loan could result in a sick child’s being treated, or a profitable micro-business, what fee is too high for that benefit?

When MFIs charge more than they need to in order to make a profit, that’s an opportunity for someone else to come in and undercut them. If no one else is coming in, that implies that the costs and difficulty of providing credit in an area may be higher than they appear to an outsider. For an outsider to declare profit margins “too high” strikes me as ungrounded and unproductive, especially when that outsider has not tried to provide credit for less in the same area.

Microfinance exists to improve the lives of the poor. Ideally, then, microfinance institutions would be judged by their effects on people’s lives. Instead, they’re being judged by simplistic financial metrics that crudely attempt to get at the moral uprightness of the organizations. To me that’s a very familiar situation.

I believe the ideal way to evaluate an MFI is to look directly at its impact. When this isn’t possible, proxies for client participation and satisfaction may (debatably) be appropriate. I don’t see any place for universal rules about how much interest can be charged.

One of the more difficult things to understand about the microfinance institutions we’ve investigated is the “true” interest rate they’re charging their borrowers. In July 2009, David Roodman of the Center for Global Development wrote:

It appears that many MFIs impose subtle fees that effectively raise interest rates. Some charge one-time loan origination fees. Some require borrowers to deposit a percentage of each loan amount with the MFI in a savings account that pays interest at a rate lower than that on the loan. Some overcharge for credit-life insurance bundled with the loan. Another criticized practice is to charge interest on the full loan amount even as the outstanding balance declines over the repayment cycle. Such “flat-rate” interest effectively doubles the interest rate compared to “declining-balance” interest since the average balance over the cycle is half the starting amount. Also, MFIs may also prefer to quote their rates on a monthly basis, hoping to exploit borrowers’ ignorance of how a seemingly modest 6 percent per month compounds into 100 percent per year.

Our experience with the microfinance organizations we have investigated to date suggests that these are real issues that donors should be aware of when interpreting interest rate data.

To be clear, we don’t think charging a high interest rate signifies wrongdoing on the part of a microfinance bank. High interest rates may be the best way to minimize losses and serve more people, and client participation at high interest rates may be an indicator that they are getting a service they value. We just want to note how striking the difference is between the initial “cited” interest rate donors often hear about and the “effective” interest rate taking all factors into account.

We recently evaluated a microfinance institution (MFI) in Malawi, the Microloan Foundation, as part of our process for distributing a grant to an economic empowerment charity in Sub-Saharan Africa. Its stated interest rate for its most popular loan type is 20%, but:

• 20% is the rate over the course of the 4-month loan. The “nominal Annual Percentage Rate (APR)” (a common way of stating interest rates in standardized terms and the rate which U.S. lenders are required to provide to borrowers) of this loan, with no other costs, would therefore be 60%.
• Interest is calculated as a flat rate. 20% of the whole loan amount is charged each payment, instead of 20% of the remaining loan balance. This method raises the nominal APR from 60% to 93%. On a $100 loan of this type, a borrower would pay $20 in interest compared with only $12.80 on a loan with declining balance interest.
• Payments are due every two weeks, instead of every month, so that the first payment is due only two weeks after the loan is made. Requiring 8 payments instead of 4 raises the nominal APR from 93% to 102%.
• The Microloan Foundation requires borrowers to hold 20% of the loan amount in a savings account which cannot be accessed until the loan is repaid. On a loan of $100, this requirement reduces the effective size of the loan to $80, while decreasing the effective size of the last payment due by $20 (because savings are then accessible). The savings requirement raises the APR from 102% to 149%.
• Arguably, the APR used above (i.e. the “nominal APR”) understates the interest rate people are paying because it does not take into account the compound value of interest. (Wikipedia’s entry on APR has a discussion of the relevant issues.) At relatively low interest rates, such as the interest rates we’re used to in the U.S., the “nominal APR” (what is usually reported) and the “effective APR” (the “mathematically true” interest rate) are usually very close to each other - but at the much higher interest rates charged by microfinance institutions, the “effective APR” can be considerably higher, raising the question of which one should be quoted to give Americans the best picture of what people are being charged. The Microloan Foundation’s nominal APR of 149% is equivalent to an effective APR of 326%.

Not only are these final “effective” interest rates many times bigger than the initial “20%” figure, they’re also significantly higher than would be implied by looking at MLF’s nominal gross portfolio yield according to MixMarket (93% for 2008; Mix Market defines this as interest and fees divided by the gross average loan portfolio ).

MLF doesn’t charge fees on its loans, but other microfinance institutions do, and these can cause further significant increases in the effective interest rate. (For example, adding a 5% fee to the beginning of a loan that without fees would have a nominal APR of 40% raises its nominal APR to 66%.)

From what we’ve seen, fees, compulsory savings, and the flat interest rate method seem to be fairly common among microfinance institutions. Of the 65 MFIs who had submitted a Social Performance Standards Report to Mix Market as of late 2009, 42% use the flat interest rate exclusively, 29% held compulsory savings accounts, and 72% collected fees on at least some loan products (according to their Mix Market profiles).

Understanding the true cost of credit for a borrower is important for reasons discussed previously:

• High interest rates (combined with high rates of repayment and low drop out rates) show that borrowers are willing to pay a high price for the loan, arguably implying that they value this service.
• If interest rates are low, microfinance institutions may be effectively giving cash transfers, at which point the mere fact of participation becomes less meaningful, and it becomes more important to ask whom the handouts are going to.
• On the other hand, the higher the interest rate, the more we worry about whether borrowing is good for the borrowers (and about anecdotes like this one).

A fairly new initiative (started in 2008), MFTransparency, is working to create a more open discussion about microloan pricing, and has compiled and published interest rate data for two countries, Bosnia and Cambodia. We look forward to following the progress of this initiative and to drawing on its data to inform our investigations of microfinance organizations.

We are concerned about the way repayment rates are often reported. We’ve written about this issue before, arguing that different delinquency indicators can easily be misleading and pointing to one example we found where a microfinance institution’s reported repayment rate substantially obscures the portion of its borrowers that have repaid loans.

Following the links from David Roodman’s recent post about Richard Rosenberg, we found another paper Mr. Rosenberg authored making all the same points, much better than we did. The paper is Richard Rosenberg’s. “Measuring microcredit delinquency: ratios can be harmful to your health.” CGAP Occasional Paper #3. 1999. Available online here (pdf).

Relevant quotes from Mr. Rosenberg’s paper

The importance of using the “right” delinquency measure:

MFIs use dozens of ratios to measure delinquency. Depending on which of them is being used, a “98 percent recovery rate” could describe a safe portfolio or one on the brink of meltdown. (Pg 1)

The measure we’ve been asking for seems to be equivalent to what he calls the “collection rate.”

Most of the discussion will be devoted to three broad types of delinquency indicators: (a) Collection rates measure amounts actually paid against amounts that have fallen due. (b) Arrears rates measure overdue amounts against total loan amounts. (c) Portfolio at risk rates measure the outstanding balance of loans that are not being paid on time against the outstanding balance of total loans. (Pg 2)

It’s essential to not only know which measure is being used, but precisely how an MFI calculates its version of the measure:

But the reader must be warned that there is no internationally consistent terminology for portfolio quality measures—for instance, what this paper calls a “collection rate” may be called a “recovery rate,” a “repayment rate,” or “loan recuperation” in other settings. No matter what name is used, the important point is that we can’t interpret what a measure is telling us unless we understand precisely the numerator and the denominator of the fraction. (Pg 2)

Mr. Rosenberg describes different tests to which MFIs should subject various delinquency measures to determine which is most appropriate. For GiveWell’s purposes, one of the key tests is the “smoke and mirrors” test:

Can the delinquency measure be made to look better through inappropriate rescheduling or refinancing of loans, or manipulation of accounting policies? This is our smoke and mirrors test. (Pg 3)

The practice of rescheduling and renegotiating loans:

When a borrower runs into repayment problems, an MFI will often renegotiate the loan, either rescheduling it (that is, stretching out its original payment terms) or refinancing it (that is, replacing it—even though the client hasn’t really repaid it—with a new loan to the same client). These practices complicate the process of using a collection rate to estimate an annual loan loss rate. Before exploring those complications and suggesting alternative solutions for dealing with them, the author needs to issue a warning: any reader looking for a perfect solution will be disappointed. The suggested approaches all have drawbacks. It is important to recognize that heavy use of rescheduling or refinancing can cloud the MFI’s ability to judge its loan loss rate. This is one of many reasons why renegotiation of problem loans should be kept to a minimum—some MFIs simply prohibit the practice. (Pg 10)

The strengths of PAR (”portfolio at risk”) as a measure:

The international standard for measuring bank loan delinquency is portfolio at risk (PAR). This measure compares apples with apples. Both the numerator and the denominator of the ratio are outstanding balances. The numerator is the unpaid balance of loans with late payments, while the denominator is the unpaid balance on all loans The PAR uses the same kind of denominator as an arrears rate, but its numerator captures all the amounts that are placed at increased risk by the delinquency. (Pg 13)

And its weaknesses:

Like many other delinquency measures, the PAR can be distorted by improper handling of renegotiated loans. MFIs sometimes reschedule—that is, amend the terms of—a problem loan, capitalizing unpaid interest and set- ting a new, longer repayment schedule. Or they may refinance a problem loan, issuing the client a new loan whose proceeds are used to pay off the old one. In both cases the delinquency is eliminated as a legal matter, but the resulting loan is clearly at higher risk than a normal loan. Thus a PAR report must age renegotiated loans separately, and provision such loans more aggressively. If this is not done, the PAR is subject to smoke and mirrors distortion: management can be tempted to give its portfolio an artificial facelift by inappropriate renegotiation. (Pg 16)

PAR can also be misleading in a situation where an MFI is growing rapidly (a key argument of our past posts):

Another potential distortion in PAR measures is worth mentioning. Arguably the PAR denominator should include only loans on which at least one payment has fallen due, so that late loans in the numerator are compared only to loans that have had a chance to be late. Nevertheless, it is customary to use the total outstanding loan balance for the denominator. The distortion involved is usually not large for MFIs, because the period before the first payment is a small fraction of the life of their loans. For instance, for a stable portfolio of loans paid in 16 weekly installments with no grace period, a PAR of 5.0 percent measured with the customary denominator (total outstanding portfolio) would rise only to 5.3 percent using the more precise denominator (excluding loans on which no payment has yet come due.) However, if a portfolio is growing very fast, or if there is a grace period or other long interval before the first payment is due, then the customary PAR denominator can seriously understate risk. Pg 17

Table 6 on Pg 19 summarizes the strengths of weaknesses of different measures:

Why is this important?

Given how complicated this all is, we think that MFIs need to be clear and transparent about (a) which measures they use and (b) precisely how they calculate them.

However, this isn’t the case. For example, we aren’t confident that most MFIs normally report rescheduled and renegotiated loans as at-risk in PAR measures.

On the one hand, Commenter Ben writes, “Best practice is to treat all loans that have been rescheduled as PAR.” (This is consistent with MixMarket’s glossary, which indicates that, “[A PAR measure] also includes loans that have been restructured or rescheduled.”

Nevertheless, “best practice” may not correlate with “in practice.”

• This Kiva document (its “Partnership Application”) is explicit in the definition of PAR 30: “The value of loans outstanding that have one or more repayments past due more than 30 days. This includes the entire unpaid balance of the loan, including both past due and future installments, but not accrued interest or renegotiated loans.” (emphasis mine) Note that, to Kiva’s credit, it explicitly asks for renegotiated loans separately in the application.
• As Holden recently commented, “At least one MFI has indicated to us that it does not report [renegotiated loans in its PAR measures].”

The definition you read today isn’t necessarily the one that MFIs are using.

What measure do we use and why?

We’ve written before that our preferred measure is what the paper discussed above calls the collection rate. While the collection rate measure fails to provide a warning to MFIs that their portfolio is in danger, it is the strongest on Mr. Rosenberg’s “Bottom-line” test because it simply and clearly measures failed repayments. It’s therefore less susceptible to obfuscation and manipulation.

For GiveWell’s purposes, we need a delinquency measure that most clearly reports borrowers’ situations. While PAR measures provide information, it’s clear that PAR measures are more valuable to evaluating the risk of an MFI’s portfolio, which while relevant is not our key concern.

We’ve thought and written a lot about microfinance lately. As of now, here’s where we stand.

What microfinance is and isn’t

First, it’s important to recognize that most of what you’ve heard about microfinance is false. It isn’t primarily about funding business expansion.. It isn’t a “proven solution” to poverty. And it doesn’t leverage your donation far more than other options.

Rather, we think of microfinance as a way to help people with low, volatile incomes manage their financial lives, an idea that is well argued in the recent Portfolios of the Poor study. This study implies that microfinance is really about providing one more option for borrowing rather than the only way to borrow, and that the borrowing is continual rather than “one crucial loan to escape poverty” - more like a credit card than a business investment. (This would explain why “graduation” from microlending programs appears rare).

What to look for

Does microfinance do good? It depends on a lot of things.

• If loans are constantly and heavily subsidized, they can be thought of as similar to giving out cash, in which case our primary concern is that benefits reach the right people.
• On the other hand, if loans are not subsidized, a microfinance institution’s profits could be taken as a sign that it has paying customers. This in turn could be a sign that it is providing empowerment.

With the latter goal (which seems to be the more common one), there is a big question about what role donations can and should play. We have expressed serious concerns about mixing donations with for-profit enterprises, with the possible result that donations end up padding profits (concept; example). In addition, we worry that there are too many donations blindly chasing the microfinance “story,” with the result that donations end up disappearing into nebulous activities.

There is also a question about the extent to which loans are truly providing empowerment. There is evidence that borrowing is bad for at least some borrowers.

We have developed a set of critical questions both about microlending and microsavings, to get at the question of whether an institution is helping people. We’ve looked hard for organizations that can answer our questions.

What we’ve found

In trying to answer the above questions, we’ve become fairly pessimistic about the area of charitable microfinance in general.

• The large U.S. charities are rarely clear even about what their value-added is. To the extent that I’ve been able to see their due diligence on partners (most of which is unfortunately confidential), it seems focused on financial performance, with much weaker examination of social impact-related questions.
• Something as simple as the “repayment rate” turns out to be reported, by standard practice, in what we feel is a highly misleading way. We’ve been shocked at how hard it is even to get what we consider the “real repayment rate” out of a microfinance charity.
• We’ve also been surprised by how often people drop out of microlending programs, generally for negative reasons.
• All in all, the picture we have is of a sector that tells stories about helping people but largely measures and rewards financial performance. As a result, microfinance institutions seem to have strong incentives to maximize their scale and their profits, even if doing so isn’t good for the borrowers.

Bottom line

All in all, we would guess that microfinance as a whole has done a great deal of good, but has also probably done some harm. We are more pessimistic specifically about microfinance donations in the current environment. For the reasons outlined above, we believe that giving to an “average” or “typical” microfinance charity – or giving with an illusory “peer to peer” relationship as the extent of your due diligence – is a fairly bad bet. At the very least, it will deliver far less good, and far more potential harm, than the typical microfinance narrative suggests.

Yet we still find the basic idea of providing financial services to people with low and volatile incomes very appealing as a way to help people … if it is done in a way that stresses social impact and uses donations responsibly.

We believe that microsavings is a particularly promising area, although we haven’t found a microsavings charity we can be confident in.

We believe that the Small Enterprise Foundation is a microlending institution that is truly and appropriately focused on achieving positive social impact. We’ll be writing more about it.

I’m very concerned about the incentives for microfinance charities. As I see it, these are the things that they are “rewarded” for:

• Profitability, for obvious reasons. Any for-profit investors in microfinance institutions are presumably putting great pressure on them to produce a good bottom line. (We believe it’s not safe to use profitability as a proxy for social impact.)
• “Scale” (clients reached, loans made, etc.) and “repayment rate.” These are the numbers that are put front and center by charities like Kiva, Grameen Foundation, and Accion among others. (We have serious questions about what the “repayment rate” even means as well as whether repaid loans = people helped.)

We feel strongly that there are many more questions a microfinance institution must answer to give an idea of whether it is helping people, and worth donating to. And in theory, many others feel this as well. But looking around MixMarket shows how much more attention the “scale and profits” indicators are getting as of today.

• The frontpage cites “1,678 total MFIs reporting data” (and we have collected the data, which is overwhelmingly complete on the financial indicators front, for the 268 members of U.S. networks). Yet only 66 institutions have filed social performance reports.
• The independent ratings found on MixMarket also seem overwhelmingly focused on issues of scale and potential profitability, as opposed to social impact (for a representative example, see the Microrate documents at WISDOM’s profile).

The bottom line? It looks to us like all of the pressure that these institutions face is to maximize scale and profits, without much eye to making sure that they’re improving clients’ lives. And that could explain some eye-popping quotes from publicly available surveys of microfinance dropouts (full documents available at that link; emphasis ours below).

• “At the end of the meeting the research team watched the process of one group being pressured to complete payment. They were told that they should not leave the meeting until the money was paid. There was heated discussion among them, with members telling each other to contribute, and denying that they themselves had the money with which to contribute. There was Ush. 8,000 missing. Finally a man from another five person group provided the missing money. He told the team that he would probably get the money back from the defaulter (who was not present) and that the defaulter would ‘probably buy him a soda’.” (Uganda report pg 13)
• “The research team also found one instance where one MFI had started operations in a poor area, issued loans, experienced quite widespread problems with loan repayment and decided to withdraw the programme in its entirety. This resulted in many people’s savings being attached to repay the outstanding loans of others, and the MFI left behind a fair degree of chaos and bad feeling.” (Uganda report pg 13)
• “the treasurer said that the group’s monthly repayment (Tsh. 300,000) that was to be deposited in SEDA account had been stolen. Two days later the treasurer got very sick; he was bewitched by some of the group members. However after seeing a witchdoctor he recovered. We took him to the Ward Executive Officer; who forced him to repay the money in instalments. He did repay the money but then he was forced to leave Arusha town completely. If he had continued to stay they would have killed him through witchcraft. I lost Tsh. 4,000 through the ROSCA.” (Tanzania report pg 10)
• “Most MFI’s in Kenya have a very strong emphasis on credit but some are getting into the position where clients are virtually forced to take loans. This is partly because of the MFI’s needs to pump out loans to achieve operational sustainability and partly due to a belief that loans are good for small entrepreneurs – even if they don’t want them!” (Kenya report pg 7)
• “One of the key determinants of drop-out, often lost in the category ‘failure to repay loan’ by these studies, is the insistence by field staff that clients take loans. Irrespective of what official Head Office policy says, there is a clear understanding among most field staff that they should push out loans - often with little care for whether the clients need or can use them… Matin (1998) also notes, ‘MFI lending technology is insensitive to variations in household conditions. Most MFIs put all households on a treadmill of continuously increasing loan size and insist on a fixed repayment schedule.’” (Bangladesh report pgs 3-4)

We’ve written before about problems with the way a microfinance institution’s “repayment rate” is commonly cited. We’ve been surprised to find that most institutions do not report what most of us would think of as a “repayment rate,” i.e., the percentage of loans/dollars due that have been paid on time. Instead, they report proxies such as “portfolio at risk” that can (theoretically) be very different.

We now have an example of just how different they can be. ID-Ghana, a microfinance institution, has given us permission to post its application for funding from GiveWell. Page 2 discloses that “The write-off ratio has shot up lately because of the clean up that followed the phase out of our old loan products which proved to be inefficient and impactless,” yet the repayment ratio is reported as 99% (same paragraph). That’s because the definition of “repayment ratio” being used ignores loans that have been defaulted on and written off. Only “at-risk” (but still-on-the-books) loans lower this “repayment ratio.”

These charts show how drastic the disrepancy is, particularly from June 2009 on:

To be clear, we think ID-Ghana’s reporting is entirely consistent with standard reporting practices. To a large degree, that’s what worries us. By industry standard reporting practices, a 99% “repayment ratio” can be consistent with a 30%+ default rate - and is in this case.

This is why we’ve insisted on requesting what we call the “real repayment rate”, defined as the percentage of loans that have been paid off on time divided by the percentage of loans that have come due over a given time period (”loans” here can refer to number of loans or dollars lent).

What has shocked us is how few microfinance institutions are able to provide the real repayment rate. In fact, all of the major U.S. microfinance institutions we’ve contacted (excluding FINCA, which declined to apply for funding at all) have explicitly told us that they cannot or will not provide real repayment rates for their partners.

We’ll be writing more about the Small Enterprise Foundation, the only institution that we’ve seen be fully clear about its repayment rate.

Just because someone is repaying their loans doesn’t mean they’re benefiting from the loans.

We have given some conceptual/anecdotal support for this idea in the past, linking to David Roodman’s posts on possible “overlending” and comparing microloans to payday loans. Lately we’ve been investigating something a bit more concrete: how often, and why, do microfinance clients “drop out” of microlending programs?

The basic idea is that a client could repay a loan due to pressure (from their “lending group” or the microfinance institution), making sacrifices or borrowing from elsewhere (such as moneylenders) to do so. We would expect such clients to show up as “repayers” while not necessarily staying in the program for more loans.

Our observations (details and full sources below):

• Dropout rates appear substantial, averaging 28% and often exceeding 40%, among institutions that publicly report them (via MixMarket).
• Survey data on why clients drop out is limited, but what we’ve seen suggests that “graduation” (i.e., moving to better sources of credit or no longer needing credit) is not a common reason for dropping out. Business failure and dissatisfaction with the group/staff/institution are common reasons.
• A high dropout rate can be consistent with reasonably good reported client satisfaction.

Dropout rates

We have gone through all microfinance institutions that report “social performance reports” on MixMarket (you can see the complete list of institutions, with their publicly posted reports, at MixMarket’s social performance report section) and collected the data into this spreadsheet (XLS). (Note that dropout rates are not on the list of “standard” indicators and are not reported by all MixMarket participants, but are included in “social performance reports.”) Here’s a summary of the 60 institutions that report dropout rates:

Taking the average across all 60, weighted by number of clients, yields an “average” dropout rate of 28%. Details here (XLS). That implies that in a given year, 28 out of 100 clients become non-clients (see the “social performance reports” for the details of the calculation).

Reasons for dropping out

We don’t know of any comprehensive studies of the reasons clients drop out, but in the process of searching for an outstanding microfinance institution, we have encountered several small-scale surveys. We have posted the non-confidential dropout surveys along with a summary in Excel, and hope to clear a couple more in the future (they are broadly consistent with the summary below).

The Bangladesh study specifically states that “One of the reasons that is notable by its almost complete absence from these listings of grounds for drop-out is ‘graduation’” (pg 4). The rest of the studies give the same picture: “graduation” (i.e., the idea that clients now no longer need microfinance because they can access better sources of credit and/or do not need credit) is not cited as a significant factor in any of them, except in the Uganda study (which does not state how common this factor is, but cites it as a factor specifically for “Relatively Well-off drop-outs” (pg iii)).

Business failure is a commonly cited factor (37-58% of clients cite this factor, in the studies that report numbers - see Excel summary). Issues with the “lending group,” the organization or its staff are the other most common factors. The Tanzania study cites “The inability of clients to cope with the rigid MFI policies and procedures” (pg 9) and also vividly describes a group conflict in which the treasurer claimed funds had been stolen (pg 10). The Bangladesh study states that “One of the key determinants of drop-out … is the insistence by field staff that clients take loans” (pg 3-4).

Client satisfaction

The LAPO study (see previous discussion of LAPO, Kiva’s largest partner) looks not only at the reasons for dropping out, but at overall reported client satisfaction.

The former figures seem cause for concern: there is a dropout rate of around 25% (estimated from graph on pg 7) and reasons given include “poor business performance” (applying to 24.2% of dropouts), “Burden of paying for others who had defaulted” (29.5%), and “the attitude of some staff” (cited as a major factor but without quantification). But overall, reported satisfaction looks reasonably strong:

We feel these numbers should be taken with a grain of salt, since it seems possible to us that clients could have felt pressure to report positive experiences. But the numbers do serve as a reminder that microfinance institutions have many clients who are (apparently) happy repeat customers.

Bottom line

Most microfinance institutions don’t appear to publicly report dropout rates, much less the reasons for dropping out (this observation based on the small percentage of MixMarket participants who have shared social performance reports). Those that do are likely to have more encouraging numbers than the others, and yet their numbers seem to leave substantial room for concern. Clients seem to drop out, for overwhelmingly negative reasons, at rates averaging 28% and often exceeding 40%.

We don’t mean to overfocus on the negative here. Microfinance institutions could be providing valuable services for many people, and we wouldn’t want donors to stay away from an activity that’s doing good overall even if it is doing damage to some.

But it does seem that the more we dig through the information on microfinance, the less it resembles the stories commonly told about it. Making loans can do good or harm. We feel strongly that people donating money to microfinance institutions should be asking for substantial due diligence - not anecdotes and pictures, and not the commonly cited, misleading metrics like “repayment rate,” but systematically collected information that gets at services’ actual impact on clients.