A few disclosures regarding this book:

• Innovations for Poverty Action is a client of Sona Partners, and Tim Ogden of Sona Partners sits on our Board.
• We are currently considering Innovations for Poverty Action for a recommendation.
• We sent the authors of the book an early draft of this review and have made modifications after some back-and-forth.

I recently read More than Good Intentions, a book by Dean Karlan and Jacob Appel of Innovations for Poverty Action (which we’ve written positive things about before).

• I thought the book was excellent, and I highly recommend it to anyone interested in effective giving and in development economics. I especially recommend it to anyone interested in microfinance – it has the best extended discussion of microfinance that I’ve seen.
• I am a bit concerned that the book doesn’t adequately capture the distinction between “exciting ideas with some preliminary evidence behind them” and “the most proven interventions in aid.” I feel that programs falling in the latter category are largely not discussed in the book, and that this fact may be unclear to donors unfamiliar with the area.
• The closing chapter of the book points readers to the Proven Impact Fund, which we understand is still a work in progress. After some discussion with Innovations for Poverty Action, we have determined that the Proven Impact Fund is unlikely to receive a GiveWell recommendation until and unless it becomes more established and has a track record. (We will be publishing the details of our views on the Proven Impact Fund at another time.) We are, however, evaluating Innovations for Poverty Action itself as a vehicle for donors to contribute to more high-quality research on aid.

Excellent discussion of microfinance and other topics

The book opens with a multi-chapter discussion of microfinance. Dean Karlan has been a leader in high-quality studies of microfinance, many of which have debunked the myths surrounding it, and I find his discussion of it to be the best I know of: engaging, educational, and sensible in its interpretation of the information we have.

The discussion centers around the idea – often lost in the stylized stories around microfinance – that different people have different needs, and that loans can be helpful to some and harmful to others. Karlan and Zinman’s study on consumer loans in the Philippines makes a strong case that when the timing is right, a loan can be very helpful even at a high interest rate. But the book also discusses people like Erlyn, whose business had a limited need for capital, such that the large long-term loans associated with her microfinance institution largely served to put her deeper into debt. The work of Karlan and others has shown how harmful many of the restrictions and rules of traditional microfinance – often based on romanticized notions of poor entrepreneurs – can be.

After discussing microfinance, More than Good Intentions goes through research on how to help the poor in other ways, from improving access to health services to raising attendance in schools. I was familiar with most of these studies prior to reading the book, but the book adds helpful context on the background and motivation for the various research projects.

One of the consistent themes of the research is the tradeoff between (a) giving clients the freedom and power to make their own choices; (b) finding “nudges” that can help people act more strongly in their self-interests. Much of the microlending research suggests that more freedom and discretion is better, but in other areas, clients seem to benefit from some restrictions – and in some cases the restrictions are even self-imposed (as with the SEED program, in which clients open savings accounts that can’t be withdrawn from until they save a certain amount).

It does seem to me that the set of investigations discussed in the book is missing an important piece. The studies cited generally compare one version of a charity’s program to another, or they compare it to “no intervention”, rather than compare a program to simply handing out cash equivalent to the program’s costs. To us this seems the most appropriate way to ethically and reasonably evaluate the value of programs: put them up against the simplest, most direct, and most empowering/discretionary of all interventions, a cash gift.

We understand that there have been studies in this field that do investigate unconditional cash transfers vs. other interventions; we also understand that funders are not always willing to finance a comparison to cash transfers as opposed to “no treatment.” Still, the fact that microfinance programs, free school uniform programs, etc. have always been compared to “no treatment” rather than “cash” seems suboptimal to us.

I also want to highlight the following anecdote from the beginning of the book, as a reminder of what low-quality evaluation (even at the largest and best-known organizations) can look like:

The first “impact evaluation” I ever saw of microcredit made my stomach ache. It was clearly intended to generate pretty numbers for a brochure to donors – not to determine whether something was really working. It asked clients something like, ‘You are eating better now, compared to before you joined FINCA, yes?’ … What I have since learned is that FINCA was doing just as much to measure its impact as anyone else. Which is to say, very little.

“Proven” vs. “exciting”

The book overall covers a lot of ground but doesn’t cover all the ground we would have liked it to, given its ambitions to encourage people to give to proven, effective interventions. This is best illustrated by the final section of the book, which aims to “leave you with seven ideas that have me [authors] particularly excited.”

The seven ideas are microsavings, reminders to save, prepaid fertilizer sales, deworming, remedial education in small groups, chlorine dispensers for clean water, and commitment devices. (The list of highlighted ideas has since been updated, as noted below.) I feel very strongly that this list (and its update) do not include the programs with the best track records in international aid. In my view, several proven health interventions – most notably immunization campaigns, distribution of insecticide-treated nets, and tuberculosis case finding and treatment – all have far better and more robust track records than any of the seven. The reasons I feel this way have to do with the distinction between micro evidence and macro evidence: the ideas in More than Good Intentions are all primarily supported by micro evidence (programs that took place on a small scale and were monitored and evaluated by academics, with the ultimate proof of effectiveness consisting in numerical differences between treatment and control groups) whereas the ideas I have listed are supported by both (not just randomized controlled trials, but also country-level success stories where a scaled-up program achieved an impact that is easy to see outside the confines of a study). I may elaborate on why this distinction is important to me in a future post.

I’m not sure that the authors of More than Good Intentions would disagree. So why does the book focus on relatively new ideas and give less attention to programs such as traditional immunization campaigns? I think there is a perfectly reasonable explanation:the authors are particularly excited about interventions that they and their colleagues have helped prove the promise of, as opposed to interventions like immunization campaigns that have been recognized as effective for decades.

There’s nothing wrong with being particularly excited about insights that are particularly new, counterintuitive, or simply a source of personal pride. However, donors need to recognize the difference between “exciting new ideas worth learning more about” and “the programs with the best track records,” and I am not sure that they will come away from More than Good Intentions with this distinction clear to them.

The closing chapter of the book points readers to the Proven Impact Fund, which we understand is still a work in progress (and unlike the list of ideas in the book, it includes interventions related to bednets and immunizations – though not the same interventions that I feel are most strongly associated with macro-level successes). After some discussion with Innovations for Poverty Action, we have determined that the Proven Impact Fund is unlikely to receive a GiveWell recommendation until and unless it becomes more established and has a track record. (We will be publishing the details of our views on the Proven Impact Fund at another time.) We are, however, evaluating Innovations for Poverty Action itself as a vehicle for donors to contribute to more high-quality research on aid.

In the meantime, we highly recommend More than Good Intentions as one of the most sophisticated discussions we’ve seen of microfinance and other fronts in the fight against poverty.