Phil Cubeta’s recent post about payday loans got me thinking about our choice to grant a microfinance organization in our Global Poverty cause.

We decided to grant Opportunity International for two reasons:

• The belief that there is likely a significant shortage of access to credit in the developing world.
• The very fact that someone repays a loan with interest likely demonstrates that the loan is used for something that is likely life-improving.

But, doesn’t the same analysis apply to payday loans? I’d bet that there’s a similar lack of credit for very small loans for borrowers with questionable credit-worthiness. And the very fact that lenders run this business likely indicates that borrowers are consistently paying back their loans, even at exorbitant interest rates (400-1000% annualized, according to the Center for Public Policy Research). The same logic that says microfinance is helping people would seem to imply that payday loans are as well.

On the other hand, it’s also possible that many borrowers are only able to repay their loans by taking out another loan – that what we’re witnessing is not a group of people getting back on their feet, but a group of people getting caught in a cycle of debt. Note that this could be numerically consistent with very high (~95%) repayment rates, the statistics commonly cited by microfinance organizations to illustrate their effectiveness in helping people – someone who borrows to pay off another loan 19 times, before finally defaulting, has a 95% repayment rate.

We’re left with two plausible yet conflicting hypotheses about the way in which the practice of making small loans at relatively high interest rates affects those in need. In one case, those in need access much needed credit (albeit at high interests rate) which allows them to weather a difficult financial period and potentially pull themselves out of poverty. In the other, those in need borrow and ultimately find themselves in a debt trap, borrowing more to repay previous loans.

We’ve generally been very frustrated with how little information we’ve been able to get on microfinance operations – who is borrowing, what they’re using the loans for, what their standard of living is, and what happens to that standard of living over time. Without this kind of information, we’re still only guessing at whether microfinance organizations and payday loan operations are helping people pull themselves out of poverty, or simply helping them get caught in cycles of debt.

We granted Population Services International in our Saving Lives in Africa cause, focusing largely on their condom and bednet distribution and promotion programs. Soon after we cut the check, I got into a conversation with someone who questioned whether bednets are effective at all. He pointed me to a paper, “Combating malaria morbidity and mortality by reducing transmission,” which questions whether bednets are likely to have a long-term impact on malaria mortality.

Bednets reduce the number of bites that people receive from malaria-carrying mosquitoes, which in turn reduces incidence of malaria. This has two effects: (a) it reduces malaria mortality; but (b) it also reduces acquired immunity to malaria. The authors of this paper found, empirically, that areas that had lower rates of bites had similar rates of malaria mortality but a different age distribution of deaths. In places with high bite-rates, malaria mortality was restricted to children; in places with low bite-rates, adults died as well. The authors hypothesize that bednet programs may only shift the age at which people die from malaria, and have no impact on the total number of deaths from malaria.

This doesn’t mean bednets are useless. First of all, the analysis in the paper isn’t general enough to cover every case – there could be population configurations such that protecting people during their weakest years is a net benefit. There could be regions with high enough bite-rates that nets merely reduce the number of bites without affecting immunity. Also, even if total morbidity and mortality don’t change, there is definitely value in people living longer before they die. But, if it’s true that reducing mosquito bites can matter much less over the long term than over the short term, this is important when weighing the value of bednets. This paper really highlights what systematic analysis and measuring can do – and what you can miss by skimping on them.

At the moment, we’re still working on our education cause; bednets are something we’re thinking about, not devoting our time to. But if we revisit the cause of aid to Africa next year, we plan to incorporate this research into our analysis, and possibly change our recommendations. We don’t expect our decisions to be perfect; they’re merely our “best bet” given what we know. Enabling anyone to provide feedback on our grant decisions, which refines (or contradicts) our previous understanding, is precisely why we think a public discussion of what charities do and how well it works is necessary.

Measuring the effect of aid on people’s lives can be difficult, and may never be perfect – but we believe that it can be done, and has been done, both rigorously and practically. Examples can be found on three sites devoted to conducting and/or promoting rigorous evaluation of social programs: Poverty Action Lab, Innovators for Poverty Action, and the Coalition for Evidence-based Policy.

These sites focus on examining and promoting particular sorts of programs (both in policy and charity), rather than on recommendations for donors; we’re working on determining how much of their information can be used to help with a donation decision. As it stands, though, all three provide good and plentiful examples of of how evaluations can be performed that are practical, rigorous, and ultimately capable of advancing our knowledge of what is likely to work. If you know of more sources of such evaluations, please share.

Developing world

The Abdul Latif Jameel Poverty Action Lab and Innovations for Poverty Action are both devoted to researching interventions aimed at alleviating developing-world poverty.

Both focus explicitly on conducting randomized controlled trials – studies that, in a nutshell, incorporate a “lottery” aspect into choosing a program’s clients, and compare those who were randomly chosen to participate to those who were randomly chosen not to. Differences between the two groups can often be attributed with reasonable confidence to the program itself, without many of the concerns over bias that can cloud the results of other kinds of studies.

Both of these organizations make their publications freely available, directly from their website. The rigor and availability of these studies makes it possible for a casual user to learn a great deal about what is likely to improve people’s lives in the developing world (for example, J-PAL provides a good deal of evidence for the positive potential of deworming programs).

U.S.

We’ve known about the above resources for a while, but only recently learned of the Coalition for Evidence-Based Policy, which has collected an impressive set of information on what works in developed-world causes such as child care, education, and employment. Like the resources above, this website focuses on studies using a randomized controlled trial approach.

Many of these studies provide direct evidence of particular charities’ effectiveness, and some of the others are useful in setting a general approach (for example, two of the studies they list – which we found through other methods – are cited in our earlier writeup on early childhood care, and influenced the way we think about this issue).

We plan to look closely at the studies on this website; we believe it has provided a valuable service in collecting some of the more rigorous evidence on what works, and that we’ll be able to use it to learn a great deal about how to accomplish as much good as possible with charitable donations. We’ll be sharing our developing thoughts as we go through its materials.