David Roodman is a research fellow at the Center for Global Development and author of David Roodman’s Microfinance Open Book Blog, where he his sharing his notes on (and content of) a book on microfinance.

We have consistently found his discussions to be evenhanded, thorough, and clear, especially in terms of what should and shouldn’t be considered evidence of impact. (See, for example, his article on evidence for microfinance’s impact, as well as his accessible discussion of research on the “macro-level” effects of international aid).

I spoke with him about what to look for in a microfinance charity. As we do, Mr. Roodman appears to see microfinance less as a tool to expand businesses than as a basic tool of empowerment that helps people manage their financial lives. He sees risks in lending – even when repayment rates are high (in fact there may be such a thing as “too high” repayment rates) and is interested in greater expansion of services to help the poor not just borrow but save money.

A rough transcript (with minor edits after the fact) follows.

---

GiveWell: What would you look for in a microfinance organization if you were trying to find one to donate to?

David Roodman: I would look for an organization that has a track record in innovation and scaling up successes. I would also look for a learning organization.

Microcredit has gone very far and has much farther to go. It also has certain risks which are pretty intuitive to us these days. And so a well-recognized challenge is, “how can we push ahead more on other financial services, especially savings?”

I would be interested in using my money to support the creation of institutions that can hold people’s savings and earn their trust. That attracts me because it doesn’t create the same risks that you have with credit. But it can be less attractive for a donor because what’s so attractive about credit is that you can see that you’re giving your money directly to beneficiaries; the middleman almost disappears. With savings it’s very different because “your” money isn’t going to clients – it’s about building an institution that can take other people’s money and hold it. It could be that your money is being used for very prosaic things.

GiveWell: Isn’t it a similar case with loans? The loans themselves are usually getting paid back with interest, but a lot of the costs of making loans are in building institutions. Even if you earmark your money “for loans,” it could be fungible with an organization’s other funds.

David Roodman: Roughly speaking, sure. Money is fungible, and so in most microcredit organizations it’s hard to pin down exactly what support is being used for.

GiveWell: You said you’d look for an organization that has innovated and scaled up successes. What exactly do you mean by success? You’ve written that success in the sense of social impact can be very hard to evaluate and there aren’t any MFIs that have conclusively demonstrated that kind of success.

David Roodman: This raises a couple of big issues about what you mean when you say something works. In the really big picture most poverty reduction has occurred as part of an economic growth process, and economic growth is not just “things getting bigger” – it’s really a societal transformation. There was a commission funded by the World Bank to write on the causes of economic growth, and the commission more or less concluded that no one knows the right recipe [http://growthcommission.org].

That means that, by and large, we don’t know how to explain most poverty reduction. We don’t know how to engineer all the poverty reduction that we want. If we don’t know what causes development, how do we act? That’s the question you’re confronting directly and very admirably. One approach is to focus on an activity where the benefit is immediate and powerful, as with vaccinations or agricultural technology.

I sometimes resort to theories about how development has happened. One of them is Amartya Sen on freedom as development: the essence of development is increasing control over one’s circumstances. From this point of view democracy is development; higher education is development; these different kinds of freedom are mutually supporting. If you can give people some kind of empowerment in any one aspect of their lives – whether education, health or income – it will probably spill over into other aspects of their lives. So even though I can’t prove to you that microfinance is raising growth rates 10 years later, if I can show you that it is giving people more control over how they manage money, there are some reasonable grounds for hope that it will contribute to these other goals.

For me, economic empowerment is a broader notion than you’re using right now. You’re free to define terms however you want. If you want to define the term literally, it’s about power over one’s circumstances. One thing I got from Portfolios of the Poor and its spiritual predecessor The Poor and Their Money is that the income of the very poor is not just low but unpredictable and volatile. And on the spending side they’ve got more uninsured risks. Their economic lives are just far less predictable.

Arguably the poor need financial services more than the rich. Financial services are inherently about that. In that sense, savings and credit and insurance are inherently about economic empowerment. It’s not the kind of economic empowerment where the person rises out of poverty through microentrepreneurship, but it gives them some more control over their lives.

The other theory is about development as institution building. What has made the US such a rich country over the past 200 years is its capacity for the development of new institutions. We don’t get stuck in a rut for very long. What’s most important to focus on in microfinance is building long-lasting, large, innovative institutions that are hopefully competing with one another and that are therefore somewhat accountable to the customer, that are into the development of new ideas, and that are upending established ways of doing things.

I think it’s important to think through these kinds of stories of how development happens, and there may be others. There’s the “private sector is the key” story. William Easterly says we should look for “searchers,” people looking for something that works. He concludes that Muhammad Yunus is such a searcher.

At the end of the day it really is hard for one person coming from the outside with a small amount of money to permanently reduce people’s poverty. Building innovative self-sustaining institutions is arguably pretty good by the standards of most public and private aid.

GiveWell: Microfinance charities often point to traditional metrics such as the repayment rate, average loan size, and number of loans made as evidence that they’re having impact. Do you feel these metrics, by themselves, constitute strong evidence of impact?

David Roodman: In the case of microcredit, we need more information. We need more information about the people who drop out. That’s a very hard thing to pin down. How many people are leaving the program each year, and why are they leaving? I don’t know if that can be pinned down.

MixMarket reports portfolio-at-risk figures, which measure default. But there may be such a thing as too high a repayment rate. It can indicate that the creditor has so much power over the borrower that they’re able to extract full repayments under dire circumstances. If you were to look at similar numbers for American banks, repayment rates would be a bit lower.

With savings and insurance, I’d be less concerned about judging by the sorts of numbers you usually see. Still, there are some questions there: for example, there might be a system that looks good on paper, but perhaps on the ground there’s so much paperwork that things are never processed or that savings are embezzled.

GiveWell: Where does the goal of profitability fit in? Is profitability or sustainability evidence that an institution is having a positive impact on people’s lives?

David Roodman: In general, profitability seems like a good thing. It means that donations are leveraged.

On the other hand, there’s always the question of whether microfinance institutions are charging too much. There’s research suggesting that if you charge even a tiny sum for a bednet that that will reduce use significantly—in other words, it might be a good thing for the bednet business to take a loss covered by subsidies. But credit is something people seem pretty willing to pay for, so the market test is more appropriate

It is case by case. When do we trust people to exercise good judgment about what they should be spending their money on? Cigarette companies are profitable.

I have somewhat lowered expectations about what you can do with microfinance, relative to the claims that are often made about it, and I’m largely interested in giving people more control over their lives. But if I’m going to give people more control over their lives, then I want to know who these people are. And it’s hard to get a lot of information about that.

Procredit is a group of banks operating in 20-30 countries. The headquarters are in Germany; the group does both savings and credit; it operates in some pretty difficult places like the DRC; it’s very much in the business of finance for ordinary people. It may not be reaching the poorest of the poor, but its philosophy is that the best way to make a contribution is to build self-sustaining businesslike banks that do both savings and credit.

GiveWell: We’ve found that we often can’t get much specific information about the demographics of clients, but perhaps there are times when it can safely be assumed that clients are low-income based on the region. Do you think that’s true?

David Roodman: If you’ve got an organization in Laos, you can figure that the people you’re reaching are pretty low-income. The same is true for most of Africa outside of South Africa. If they’re reaching some respectable scale, if they’re not just getting it to the top landowners in the village.

That said, Congress passed a law here some years ago requiring that half of U.S. microfinance aid reach people below the poverty line of about $1/day (purchasing power parity adjusted – a very low poverty line). That created quite a large demand for information about the demographics of borrowers so I think that there is more and more of that kind of information.

GiveWell: Your thoughts on other areas of economic empowerment focused charity, such as agricultural programs?

David Roodman: I don’t know much about these areas. I worry about when a charity starts deciding what recipients need – why not just give them the money and let them decide? This always should be a question in the background, and does actually point to virtue of microfinance, which helps people manage money for their own goals.

One reason we don’t give them the money is that we hope there’s some leverage point where we can multiply the value of our charity. But as soon as you get into distribution of goods, you’re not multiplying value of your charity.

GiveWell: What about development of agricultural technology?

David Roodman: This is an area I’m not very familiar with. In Africa, there isn’t a good track record. If you look at the developing world as a whole, there have been huge gains in crop yields, in part due to philanthropy. Not without negative side effects.

GiveWell: Is there any particular area of economic empowerment focused charity, outside of microfinance, that you find particularly promising?

David Roodman: Support for medium enterprises in intriguing because it is not looked at much. I haven’t looked at it much. Enterprises that are a bit larger are important. In the end, most very poor entrepreneurs are entrepreneurs out of necessity. They want jobs. And somewhat larger enterprises create them as they grow.

GiveWell: The way we’re leaning is that if we can tell that an institution is working with people in need, from there what we’d most like to see is a strong track record of starting self-sustaining initiatives; another thing we’d like to see is social impact over time (even if not demonstrated with great rigor). Probably third on our list is a straight transfer of wealth to demonstrably low-income people.

That all sounds very reasonable.

I see Yunus as a Henry Ford type figure. He didn’t really invent microfinance, but what he did amazingly well was experiment and develop business solutions to the problem of how an institution can provide financial services to the poor without losing its shirt. He created a large-scale business. A lot of what characterizes modern microcredit comes from wanting to control costs. Part of what group credit is about is shifting a lot of the bankers’ jobs onto the borrowers, including collecting info about who the borrowers are and how well-off they are. It’s a characteristic of the business to not know clients very well. Asking for information about clients means asking for a business model that has higher transaction costs and must therefore charge higher interest rates or serve richer people. Perhaps charitable funds can help cover that cost, but it’s a difficult tradeoff.

GiveWell: One of our leading contenders for a grant right now is the Village Enterprise Fund. It focuses not on loans but on grants, along with general advice and mentoring. It serves extremely low-income people (below the poverty line) and regularly and systematically collects data about their standards of living (it doesn’t attempt to gauge “income” but instead looks at quality-of-life factors such as access to clean water.) It has a recent study showing major changes in standards of living over time, although there is no comparison group.

David Roodman: Sounds reasonable. It sounds like a high transaction cost operation.

I find it refreshing that the focus is on grants. With all the talk of credit and savings, giving people money and oversight and advice is a refreshing change.

There’s always potential for selection bias in a study like the one you mentioned. It’s possible that clients come to the program as they’re about to succeed for other reasons.

The kind of standard-of-living assessment they’re doing is increasingly common in microfinance as well. In fact I think it may even be the norm. There’s a guy named Mark Schreiner who has developed scoring formulas that people can quickly apply to estimate “income” based on other quality of life measures.

GiveWell: Do you know where to get that data?

David Roodman: Check microfinance.com. The reports are called Poverty Scorecards.

GiveWell: Who else would you suggest that we talk to?

David Roodman: One person I would recommend is a colleague of mine, Vijaya Ramachandran, who has done a lot of primary research on small enterprises in Africa and what helps them grow or prevents them from growing. Another person who comes to mind is Dennis Whittle of GlobalGiving, which is largely about “letting the market choose” in philanthropy.

In general, it sounds like you’re looking for something that you can have high confidence in. There’s another approach one can take, the venture capital approach of taking risks with big potential payoffs. That might make more sense for much larger pools of funds, though, as opposed to giving away $250,000 and making recommendations to individual donors.

GiveWell: In addition to microfinance, you seem to be interested in the general topic of methodology in the social sciences. You’ve written a couple of papers that were very focused on methodology, which I liked. So I thought I would get your thoughts on a general pattern I’ve noticed in my reaction to impact studies.

I’ve found that whenever I’m looking at a case-control study that is only based on a single point in time – comparing clients of a program to non-clients, at a single time – I am almost always extremely skeptical, very worried about selection bias, and it’s very hard for the paper to make me accept the result. However, when I look at a study that examines changes over time, when the changes are big and coincide with the implementation of the program, I’m much more likely to see this as suggestive evidence of impact – even without a control group. (The Village Enterprise Fund study, for example.) Reflecting on this pattern, I think it’s reasonable to have this bias, but what are your thoughts?

David Roodman: I’m not sure how to give a general answer. I think you understand: if there’s a modest change over time, then there could be all sorts of alternative explanations for the change. If it’s an extraordinarily large change, then yes, it becomes harder to explain away. But it still can be selection bias. Perhaps the people who were about to succeed were the ones who found the program.

There are some large-scale improvements in health that have been attributed to vaccines and to ORT. One could argue that we don’t know whether this impact is real or coincidental, but the change over time is just so huge that it seems like it has to be real.

All in all, I think you have a good point. A given person is more like himself/herself (at a different point in time) than like someone else (at the same point in time). This makes changes over time harder to explain away.

We wrote last week that the direct evidence of microfinance’s impact is less than overwhelming. However, many believe that direct evidence is not needed – that as long as microfinance institutions demonstrate high “repayment rates,” they can be assumed to be improving lives. The logic goes that if a person is able to repay their loan with interest, they must have used it productively.

We believe the repayment rate doesn’t tell you nearly enough to have confidence.

Concern 1: what does the “repayment rate” actually mean?

One of the things that has surprised us, as we’ve looked into microfinance, is that we haven’t seen a clear or consistent definition of the “repayment rate” often featured on charities’ websites.

The excellent MixMarket site is a clearinghouse for standardized information from MFIs. But looking through its glossary doesn’t turn up any statistics that are clearly equivalent to the repayment rate (i.e., the percentage of loans due that are repaid on time). The closest items appear to be:

• “Loan loss rate”: the loans written off (minus any payments recovered post writeoff) divided by the total outstanding loan balance.

• “Portfolio at risk > [XX] days ratio”: the loans that have a payment past due more than [XX] days, divided by – again – the total outstanding loan balance.

One problem with both of these statistics is that they are taken as a percentage of loans outstanding, not loans due. In other words, if one takes either of these measures as the inverse of “repayment rate,” an MFI can inflate this “repayment rate” simply by growing its loan portfolio (making loans that aren’t yet due). Consider the simplified case of an MFI that has made $1000 worth of loans that are all past due, and $2000 worth of new loans that aren’t due yet. In this case it will list a “loss rate” of only 1/3, even though all of the loans due have failed to repay. This is an especially serious concern because of how common it is for an MFI to be rapidly expanding its loan portfolio. Out of 257 MFIs we examined with the relevant data, 82 have seen their portfolio grow by 50% or more in the most recent reported year, while 21 have seen 100%+ growth.

In addition, both of these statistics rely heavily on which loans an MFI chooses to “write off.” An MFI that chooses to hold bad loans on its books could have a very low “loan loss rate”; conversely, an MFI that chooses to write off many loans could have a very low “portfolio at risk” (since a loan that is written off is no longer considered part of the portfolio).

We spoke with Accion International and asked about the figure on its website claiming a historical repayment rate of 97%. We were told that this figure is calculated via “loans paid” divided by “loans due,” which distinguishes it from both of the standardized figures listed above. However, we were also told that loans can be counted as “repaid” even if they’re restructured (i.e., the terms of the loans are changed to accommodate late or incomplete repayment). Another microfinance charity, FINCA, has told us that “portfolio at risk” (discussed above) is the metric used.

Bottom line: the “repayment rate” figure is highly subject to interpretation, unless the organization giving it is very specific about how it’s calculated.

Concern 2: is a repaid loan always a good thing?

This question has been discussed at length on David Roodman’s blog about microfinance. Some concerns:

• High repayment rates could indicate coercive collection practices – especially because the responsibility for collection is often left to “lending groups” with limited oversight from the institution.
• People may be taking out more credit – and thus paying more fees – than is in their own best interest (even if they can repay the loans).
• People may be engaging in unsustainable borrowing patterns, such as repeatedly borrowing from one MFI to pay back another or oversaturating the market with their microenterprises. Roodman’s latest post on the idea of a “microfinance bubble” concludes that there is little hard evidence that such “bubbles” are occurring, but unsustainable borrowing is a possibility.

Concern 3: who is taking out the loans?

Are the people served by MFIs extremely poor? Relatively poor? Relatively wealthy? Moneylenders themselves, taking advantage of donor subsidies?

When donations are used to offer a loan at a below-market rate, a loan may be quite functionally similar to a a cash gift, especially if funds can be re-lent informally (and studies such as Portfolios of the Poor suggest that the informal market for lending is quite active).

Note that a high repayment rate could partially be the product of wealthier clients’ repayments, even if repayments from lower-income clients are less reliable.

Beyond repayment rates

Here is some of the information we think an MFI can provide to address the concerns above:

• A clear and explicit definition of “repayment rate.” Are “loans outstanding” implicitly counted as “loans repaid” (as in the MixMarket measures we discussed) or does the rate only include loans due? Can loans be counted as repaid when they have been restructured or delayed? How common and drastic are restructurings and delays?
• Information on clients’ incomes and standard of living. How are clients selected? How poor are clients?
• Information on client turnover. How often do clients leave the program, and for what reasons? (Presumably, clients would be more likely to continue taking out loans if the loans were truly beneficial for them.)
• Rigorous impact studies, which we’ve discussed at length.