We are concerned about the way repayment rates are often reported. We’ve written about this issue before, arguing that different delinquency indicators can easily be misleading and pointing to one example we found where a microfinance institution’s reported repayment rate substantially obscures the portion of its borrowers that have repaid loans.

Following the links from David Roodman’s recent post about Richard Rosenberg, we found another paper Mr. Rosenberg authored making all the same points, much better than we did. The paper is Richard Rosenberg’s. “Measuring microcredit delinquency: ratios can be harmful to your health.” CGAP Occasional Paper #3. 1999. Available online here (pdf).

Relevant quotes from Mr. Rosenberg’s paper

The importance of using the “right” delinquency measure:

MFIs use dozens of ratios to measure delinquency. Depending on which of them is being used, a “98 percent recovery rate” could describe a safe portfolio or one on the brink of meltdown. (Pg 1)

The measure we’ve been asking for seems to be equivalent to what he calls the “collection rate.”

Most of the discussion will be devoted to three broad types of delinquency indicators: (a) Collection rates measure amounts actually paid against amounts that have fallen due. (b) Arrears rates measure overdue amounts against total loan amounts. (c) Portfolio at risk rates measure the outstanding balance of loans that are not being paid on time against the outstanding balance of total loans. (Pg 2)

It’s essential to not only know which measure is being used, but precisely how an MFI calculates its version of the measure:

But the reader must be warned that there is no internationally consistent terminology for portfolio quality measures—for instance, what this paper calls a “collection rate” may be called a “recovery rate,” a “repayment rate,” or “loan recuperation” in other settings. No matter what name is used, the important point is that we can’t interpret what a measure is telling us unless we understand precisely the numerator and the denominator of the fraction. (Pg 2)

Mr. Rosenberg describes different tests to which MFIs should subject various delinquency measures to determine which is most appropriate. For GiveWell’s purposes, one of the key tests is the “smoke and mirrors” test:

Can the delinquency measure be made to look better through inappropriate rescheduling or refinancing of loans, or manipulation of accounting policies? This is our smoke and mirrors test. (Pg 3)

The practice of rescheduling and renegotiating loans:

When a borrower runs into repayment problems, an MFI will often renegotiate the loan, either rescheduling it (that is, stretching out its original payment terms) or refinancing it (that is, replacing it—even though the client hasn’t really repaid it—with a new loan to the same client). These practices complicate the process of using a collection rate to estimate an annual loan loss rate. Before exploring those complications and suggesting alternative solutions for dealing with them, the author needs to issue a warning: any reader looking for a perfect solution will be disappointed. The suggested approaches all have drawbacks. It is important to recognize that heavy use of rescheduling or refinancing can cloud the MFI’s ability to judge its loan loss rate. This is one of many reasons why renegotiation of problem loans should be kept to a minimum—some MFIs simply prohibit the practice. (Pg 10)

The strengths of PAR (”portfolio at risk”) as a measure:

The international standard for measuring bank loan delinquency is portfolio at risk (PAR). This measure compares apples with apples. Both the numerator and the denominator of the ratio are outstanding balances. The numerator is the unpaid balance of loans with late payments, while the denominator is the unpaid balance on all loans The PAR uses the same kind of denominator as an arrears rate, but its numerator captures all the amounts that are placed at increased risk by the delinquency. (Pg 13)

And its weaknesses:

Like many other delinquency measures, the PAR can be distorted by improper handling of renegotiated loans. MFIs sometimes reschedule—that is, amend the terms of—a problem loan, capitalizing unpaid interest and set- ting a new, longer repayment schedule. Or they may refinance a problem loan, issuing the client a new loan whose proceeds are used to pay off the old one. In both cases the delinquency is eliminated as a legal matter, but the resulting loan is clearly at higher risk than a normal loan. Thus a PAR report must age renegotiated loans separately, and provision such loans more aggressively. If this is not done, the PAR is subject to smoke and mirrors distortion: management can be tempted to give its portfolio an artificial facelift by inappropriate renegotiation. (Pg 16)

PAR can also be misleading in a situation where an MFI is growing rapidly (a key argument of our past posts):

Another potential distortion in PAR measures is worth mentioning. Arguably the PAR denominator should include only loans on which at least one payment has fallen due, so that late loans in the numerator are compared only to loans that have had a chance to be late. Nevertheless, it is customary to use the total outstanding loan balance for the denominator. The distortion involved is usually not large for MFIs, because the period before the first payment is a small fraction of the life of their loans. For instance, for a stable portfolio of loans paid in 16 weekly installments with no grace period, a PAR of 5.0 percent measured with the customary denominator (total outstanding portfolio) would rise only to 5.3 percent using the more precise denominator (excluding loans on which no payment has yet come due.) However, if a portfolio is growing very fast, or if there is a grace period or other long interval before the first payment is due, then the customary PAR denominator can seriously understate risk. Pg 17

Table 6 on Pg 19 summarizes the strengths of weaknesses of different measures:

Why is this important?

Given how complicated this all is, we think that MFIs need to be clear and transparent about (a) which measures they use and (b) precisely how they calculate them.

However, this isn’t the case. For example, we aren’t confident that most MFIs normally report rescheduled and renegotiated loans as at-risk in PAR measures.

On the one hand, Commenter Ben writes, “Best practice is to treat all loans that have been rescheduled as PAR.” (This is consistent with MixMarket’s glossary, which indicates that, “[A PAR measure] also includes loans that have been restructured or rescheduled.”

Nevertheless, “best practice” may not correlate with “in practice.”

• This Kiva document (its “Partnership Application”) is explicit in the definition of PAR 30: “The value of loans outstanding that have one or more repayments past due more than 30 days. This includes the entire unpaid balance of the loan, including both past due and future installments, but not accrued interest or renegotiated loans.” (emphasis mine) Note that, to Kiva’s credit, it explicitly asks for renegotiated loans separately in the application.
• As Holden recently commented, “At least one MFI has indicated to us that it does not report [renegotiated loans in its PAR measures].”

The definition you read today isn’t necessarily the one that MFIs are using.

What measure do we use and why?

We’ve written before that our preferred measure is what the paper discussed above calls the collection rate. While the collection rate measure fails to provide a warning to MFIs that their portfolio is in danger, it is the strongest on Mr. Rosenberg’s “Bottom-line” test because it simply and clearly measures failed repayments. It’s therefore less susceptible to obfuscation and manipulation.

For GiveWell’s purposes, we need a delinquency measure that most clearly reports borrowers’ situations. While PAR measures provide information, it’s clear that PAR measures are more valuable to evaluating the risk of an MFI’s portfolio, which while relevant is not our key concern.

We’ve thought and written a lot about microfinance lately. As of now, here’s where we stand.

What microfinance is and isn’t

First, it’s important to recognize that most of what you’ve heard about microfinance is false. It isn’t primarily about funding business expansion.. It isn’t a “proven solution” to poverty. And it doesn’t leverage your donation far more than other options.

Rather, we think of microfinance as a way to help people with low, volatile incomes manage their financial lives, an idea that is well argued in the recent Portfolios of the Poor study. This study implies that microfinance is really about providing one more option for borrowing rather than the only way to borrow, and that the borrowing is continual rather than “one crucial loan to escape poverty” - more like a credit card than a business investment. (This would explain why “graduation” from microlending programs appears rare).

What to look for

Does microfinance do good? It depends on a lot of things.

• If loans are constantly and heavily subsidized, they can be thought of as similar to giving out cash, in which case our primary concern is that benefits reach the right people.
• On the other hand, if loans are not subsidized, a microfinance institution’s profits could be taken as a sign that it has paying customers. This in turn could be a sign that it is providing empowerment.

With the latter goal (which seems to be the more common one), there is a big question about what role donations can and should play. We have expressed serious concerns about mixing donations with for-profit enterprises, with the possible result that donations end up padding profits (concept; example). In addition, we worry that there are too many donations blindly chasing the microfinance “story,” with the result that donations end up disappearing into nebulous activities.

There is also a question about the extent to which loans are truly providing empowerment. There is evidence that borrowing is bad for at least some borrowers.

We have developed a set of critical questions both about microlending and microsavings, to get at the question of whether an institution is helping people. We’ve looked hard for organizations that can answer our questions.

What we’ve found

In trying to answer the above questions, we’ve become fairly pessimistic about the area of charitable microfinance in general.

• The large U.S. charities are rarely clear even about what their value-added is. To the extent that I’ve been able to see their due diligence on partners (most of which is unfortunately confidential), it seems focused on financial performance, with much weaker examination of social impact-related questions.
• Something as simple as the “repayment rate” turns out to be reported, by standard practice, in what we feel is a highly misleading way. We’ve been shocked at how hard it is even to get what we consider the “real repayment rate” out of a microfinance charity.
• We’ve also been surprised by how often people drop out of microlending programs, generally for negative reasons.
• All in all, the picture we have is of a sector that tells stories about helping people but largely measures and rewards financial performance. As a result, microfinance institutions seem to have strong incentives to maximize their scale and their profits, even if doing so isn’t good for the borrowers.

Bottom line

All in all, we would guess that microfinance as a whole has done a great deal of good, but has also probably done some harm. We are more pessimistic specifically about microfinance donations in the current environment. For the reasons outlined above, we believe that giving to an “average” or “typical” microfinance charity – or giving with an illusory “peer to peer” relationship as the extent of your due diligence – is a fairly bad bet. At the very least, it will deliver far less good, and far more potential harm, than the typical microfinance narrative suggests.

Yet we still find the basic idea of providing financial services to people with low and volatile incomes very appealing as a way to help people … if it is done in a way that stresses social impact and uses donations responsibly.

We believe that microsavings is a particularly promising area, although we haven’t found a microsavings charity we can be confident in.

We believe that the Small Enterprise Foundation is a microlending institution that is truly and appropriately focused on achieving positive social impact. We’ll be writing more about it.

I’m very concerned about the incentives for microfinance charities. As I see it, these are the things that they are “rewarded” for:

• Profitability, for obvious reasons. Any for-profit investors in microfinance institutions are presumably putting great pressure on them to produce a good bottom line. (We believe it’s not safe to use profitability as a proxy for social impact.)
• “Scale” (clients reached, loans made, etc.) and “repayment rate.” These are the numbers that are put front and center by charities like Kiva, Grameen Foundation, and Accion among others. (We have serious questions about what the “repayment rate” even means as well as whether repaid loans = people helped.)

We feel strongly that there are many more questions a microfinance institution must answer to give an idea of whether it is helping people, and worth donating to. And in theory, many others feel this as well. But looking around MixMarket shows how much more attention the “scale and profits” indicators are getting as of today.

• The frontpage cites “1,678 total MFIs reporting data” (and we have collected the data, which is overwhelmingly complete on the financial indicators front, for the 268 members of U.S. networks). Yet only 66 institutions have filed social performance reports.
• The independent ratings found on MixMarket also seem overwhelmingly focused on issues of scale and potential profitability, as opposed to social impact (for a representative example, see the Microrate documents at WISDOM’s profile).

The bottom line? It looks to us like all of the pressure that these institutions face is to maximize scale and profits, without much eye to making sure that they’re improving clients’ lives. And that could explain some eye-popping quotes from publicly available surveys of microfinance dropouts (full documents available at that link; emphasis ours below).

• “At the end of the meeting the research team watched the process of one group being pressured to complete payment. They were told that they should not leave the meeting until the money was paid. There was heated discussion among them, with members telling each other to contribute, and denying that they themselves had the money with which to contribute. There was Ush. 8,000 missing. Finally a man from another five person group provided the missing money. He told the team that he would probably get the money back from the defaulter (who was not present) and that the defaulter would ‘probably buy him a soda’.” (Uganda report pg 13)
• “The research team also found one instance where one MFI had started operations in a poor area, issued loans, experienced quite widespread problems with loan repayment and decided to withdraw the programme in its entirety. This resulted in many people’s savings being attached to repay the outstanding loans of others, and the MFI left behind a fair degree of chaos and bad feeling.” (Uganda report pg 13)
• “the treasurer said that the group’s monthly repayment (Tsh. 300,000) that was to be deposited in SEDA account had been stolen. Two days later the treasurer got very sick; he was bewitched by some of the group members. However after seeing a witchdoctor he recovered. We took him to the Ward Executive Officer; who forced him to repay the money in instalments. He did repay the money but then he was forced to leave Arusha town completely. If he had continued to stay they would have killed him through witchcraft. I lost Tsh. 4,000 through the ROSCA.” (Tanzania report pg 10)
• “Most MFI’s in Kenya have a very strong emphasis on credit but some are getting into the position where clients are virtually forced to take loans. This is partly because of the MFI’s needs to pump out loans to achieve operational sustainability and partly due to a belief that loans are good for small entrepreneurs – even if they don’t want them!” (Kenya report pg 7)
• “One of the key determinants of drop-out, often lost in the category ‘failure to repay loan’ by these studies, is the insistence by field staff that clients take loans. Irrespective of what official Head Office policy says, there is a clear understanding among most field staff that they should push out loans - often with little care for whether the clients need or can use them… Matin (1998) also notes, ‘MFI lending technology is insensitive to variations in household conditions. Most MFIs put all households on a treadmill of continuously increasing loan size and insist on a fixed repayment schedule.’” (Bangladesh report pgs 3-4)

We’ve written before about problems with the way a microfinance institution’s “repayment rate” is commonly cited. We’ve been surprised to find that most institutions do not report what most of us would think of as a “repayment rate,” i.e., the percentage of loans/dollars due that have been paid on time. Instead, they report proxies such as “portfolio at risk” that can (theoretically) be very different.

We now have an example of just how different they can be. ID-Ghana, a microfinance institution, has given us permission to post its application for funding from GiveWell. Page 2 discloses that “The write-off ratio has shot up lately because of the clean up that followed the phase out of our old loan products which proved to be inefficient and impactless,” yet the repayment ratio is reported as 99% (same paragraph). That’s because the definition of “repayment ratio” being used ignores loans that have been defaulted on and written off. Only “at-risk” (but still-on-the-books) loans lower this “repayment ratio.”

These charts show how drastic the disrepancy is, particularly from June 2009 on:

To be clear, we think ID-Ghana’s reporting is entirely consistent with standard reporting practices. To a large degree, that’s what worries us. By industry standard reporting practices, a 99% “repayment ratio” can be consistent with a 30%+ default rate - and is in this case.

This is why we’ve insisted on requesting what we call the “real repayment rate”, defined as the percentage of loans that have been paid off on time divided by the percentage of loans that have come due over a given time period (”loans” here can refer to number of loans or dollars lent).

What has shocked us is how few microfinance institutions are able to provide the real repayment rate. In fact, all of the major U.S. microfinance institutions we’ve contacted (excluding FINCA, which declined to apply for funding at all) have explicitly told us that they cannot or will not provide real repayment rates for their partners.

We’ll be writing more about the Small Enterprise Foundation, the only institution that we’ve seen be fully clear about its repayment rate.

Just because someone is repaying their loans doesn’t mean they’re benefiting from the loans.

We have given some conceptual/anecdotal support for this idea in the past, linking to David Roodman’s posts on possible “overlending” and comparing microloans to payday loans. Lately we’ve been investigating something a bit more concrete: how often, and why, do microfinance clients “drop out” of microlending programs?

The basic idea is that a client could repay a loan due to pressure (from their “lending group” or the microfinance institution), making sacrifices or borrowing from elsewhere (such as moneylenders) to do so. We would expect such clients to show up as “repayers” while not necessarily staying in the program for more loans.

Our observations (details and full sources below):

• Dropout rates appear substantial, averaging 28% and often exceeding 40%, among institutions that publicly report them (via MixMarket).
• Survey data on why clients drop out is limited, but what we’ve seen suggests that “graduation” (i.e., moving to better sources of credit or no longer needing credit) is not a common reason for dropping out. Business failure and dissatisfaction with the group/staff/institution are common reasons.
• A high dropout rate can be consistent with reasonably good reported client satisfaction.

Dropout rates

We have gone through all microfinance institutions that report “social performance reports” on MixMarket (you can see the complete list of institutions, with their publicly posted reports, at MixMarket’s social performance report section) and collected the data into this spreadsheet (XLS). (Note that dropout rates are not on the list of “standard” indicators and are not reported by all MixMarket participants, but are included in “social performance reports.”) Here’s a summary of the 60 institutions that report dropout rates:

Taking the average across all 60, weighted by number of clients, yields an “average” dropout rate of 28%. Details here (XLS). That implies that in a given year, 28 out of 100 clients become non-clients (see the “social performance reports” for the details of the calculation).

Reasons for dropping out

We don’t know of any comprehensive studies of the reasons clients drop out, but in the process of searching for an outstanding microfinance institution, we have encountered several small-scale surveys. We have posted the non-confidential dropout surveys along with a summary in Excel, and hope to clear a couple more in the future (they are broadly consistent with the summary below).

The Bangladesh study specifically states that “One of the reasons that is notable by its almost complete absence from these listings of grounds for drop-out is ‘graduation’” (pg 4). The rest of the studies give the same picture: “graduation” (i.e., the idea that clients now no longer need microfinance because they can access better sources of credit and/or do not need credit) is not cited as a significant factor in any of them, except in the Uganda study (which does not state how common this factor is, but cites it as a factor specifically for “Relatively Well-off drop-outs” (pg iii)).

Business failure is a commonly cited factor (37-58% of clients cite this factor, in the studies that report numbers - see Excel summary). Issues with the “lending group,” the organization or its staff are the other most common factors. The Tanzania study cites “The inability of clients to cope with the rigid MFI policies and procedures” (pg 9) and also vividly describes a group conflict in which the treasurer claimed funds had been stolen (pg 10). The Bangladesh study states that “One of the key determinants of drop-out … is the insistence by field staff that clients take loans” (pg 3-4).

Client satisfaction

The LAPO study (see previous discussion of LAPO, Kiva’s largest partner) looks not only at the reasons for dropping out, but at overall reported client satisfaction.

The former figures seem cause for concern: there is a dropout rate of around 25% (estimated from graph on pg 7) and reasons given include “poor business performance” (applying to 24.2% of dropouts), “Burden of paying for others who had defaulted” (29.5%), and “the attitude of some staff” (cited as a major factor but without quantification). But overall, reported satisfaction looks reasonably strong:

We feel these numbers should be taken with a grain of salt, since it seems possible to us that clients could have felt pressure to report positive experiences. But the numbers do serve as a reminder that microfinance institutions have many clients who are (apparently) happy repeat customers.

Bottom line

Most microfinance institutions don’t appear to publicly report dropout rates, much less the reasons for dropping out (this observation based on the small percentage of MixMarket participants who have shared social performance reports). Those that do are likely to have more encouraging numbers than the others, and yet their numbers seem to leave substantial room for concern. Clients seem to drop out, for overwhelmingly negative reasons, at rates averaging 28% and often exceeding 40%.

We don’t mean to overfocus on the negative here. Microfinance institutions could be providing valuable services for many people, and we wouldn’t want donors to stay away from an activity that’s doing good overall even if it is doing damage to some.

But it does seem that the more we dig through the information on microfinance, the less it resembles the stories commonly told about it. Making loans can do good or harm. We feel strongly that people donating money to microfinance institutions should be asking for substantial due diligence - not anecdotes and pictures, and not the commonly cited, misleading metrics like “repayment rate,” but systematically collected information that gets at services’ actual impact on clients.

Note: I’ve responded to the most recent batch of comments.

A lot of work has been put into estimating the “bang for your buck” in health initiatives. In the area of microfinance, though, things appear very murky.

Microfinance advocates say things like “As our clients repay the loans, the money is loaned again and again to help many more entrepreneurs. It’s giving that keeps going.” Skeptics reply that much of the cost of lending is in operating institutions, not simply loan capital. We should be able to agree that the cost-effectiveness of microlending is not literally infinite, but what’s the right ballpark? Does the impact per dollar dwarf that of health?

We can take a very rough - and very generous to microfinance - cut by looking at some global estimates by CGAP. Notes before we get to the numbers:

• We are trying to get a number that we can put alongside existing estimates of health cost-effectiveness, just to see whether the microfinance sector as a whole has a clear and large advantage in cost-effectiveness. The estimate will be extremely rough and will not apply to any given microfinance charity, but rather to the area of microfinance as a whole.
• Our estimate is essentially a “best-case scenario” for what microfinance cost-effectiveness would be if (a) there were a direct link between donations and people served (b) microfinance could reach an enormous “target population” at the same level of donation funding that’s being provided now.

CGAP looks at both dollars invested in microfinance (PDF) and people served. According to these links,

• $11.7 billion of funding went to microfinance in 2008, of which 19% - or ~$2.2 billion - was grants (not loans, not investments, not guarantees).
• There are currently between 130 and 190 million microfinance borrowers worldwide.
• CGAP implies a “target number” of borrowers: “Given that almost 3bn people live on less than two dollars a day, clearly the battle to bring financial access to as many people as possible is a very long way from being won.” I have major issues with this target - for one thing, I’m not sure that people living under $2/day should all be targets, or are the only targets, of MFIs.

A couple of ways to look at the “costs per MFI client”:

• A lot of money is spent on microfinance. $2 billion in grants is about 10% as much as the total official development aid of the U.S. government (according to the 2008 Index of Global Philanthropy (PDF)).
• We’re currently spending $12-$17 in grants alone for every MFI borrower. Of course, the grants could be paying for a lot more than borrowing (including savings), and could be made with the aim of expanding future services rather than maintaining existing ones.
• If you believe that microfinance will eventually reach the entire CGAP “target population” (or a population that size, which would be around half the population of the world) and that the current level of grants will be maintained (say, growing only at the rate that the size of the target population grows), then at the point where microfinance is reaching its entire “target population,” the grants per person reached will be about $0.75. While this figure could be overstating the costs per person served if grants eventually create self-sustaining institutions and become unnecessary, I think it is far more likely that it understates the cost because (a) those who can most practically be reached in a profitable/sustainable way are likely to be those already reached, and the hardest people to reach are more likely to require continued subsidies; (b) there is a huge amount of other investment in microfinance, and we have very little sense of the role that grants play in enabling the expansion of services; (c) 3 billion clients is an extremely ambitious goal - around 20x the number of people actually being reached today, and around half the world’s current population.

A couple of ways to think about the comparison with health:

• For $200, would you rather provide about 250 people with a year’s worth of financial services, or prevent a single death via vaccination?
• For $3.50, would you rather provide 5 people with a year’s worth of financial services or 14 children with deworming drugs, leading to one extra year of school attendance?

I would answer both of these questions mostly with a shrug. Certainly, under this extremely generous estimate of what microfinance could cost, it is “competitive” with the best health programs.

But this is assuming that all of that money going to microfinance is going to eventually succeed in reaching half the world, and also making the even bigger assumption that grants are the key factor. We think it’s very possible that much of microfinance’s reach has very little, or even literally nothing, to do with charitable support. (The less generous cost-effectiveness estimate of $12-$17 is fairly clearly not competitive with the best health programs: compare 12-17 person-years of financial services vs. 1 life saved, or 1 person-year of financial services vs. 3-5 person-years of extra school attendance due to improved health.)

Bottom line: we don’t see cost-effectiveness or “multiplying the impact of your dollar” as a strong argument for funding microfinance over health, on a general sector-level basis. This is the case even under the most generous model of the microfinance figures we’ve come up with.