A donor wrote to us recently with a question (paraphrased):

Is it better to help people who cannot currently live a decent life or to help those who might be better off to start with but have more potential to live enjoyable and meaningful lives?

Ideally the answer to this question would be “let’s help everyone get to the point where they can live enjoyable and meaningful lives.” Realistically, though, resources available for charity are not unlimited and there are trade-offs involved in choosing where to give.

Recently, we’ve been internally debating the question above, which said another way is: “If you had similarly strong opportunities to help different groups, who do you help to do the most good?” and we have some differences of opinion. For example:

• Holden believes that the ultimate goal, the highest moral imperative, is to achieve and help others achieve their full potential, i.e. help them do something great with their lives.
• I believe that the ultimate goal is to reduce suffering and create happiness.

Holden and I agree, though, that these goals often lead to similar intermediate goals:

1. Fulfill people’s basic needs
2. Create environments in which there are plenty of good opportunities and people can choose for themselves what is best.

Where we tend to differ is the populations we are most concerned with helping. Given the choice, Holden would prefer to help those with high potential to do great things with their lives (one frequent example that comes up is young people in fast-growing, democratic India), whereas I most want to help those in abject poverty who suffer from hunger, disease, and uncertainty (this tends to be people in isolated, rural areas in developing countries).

That being said, Holden and I agree that within our current known charitable options, we choose to give to VillageReach. VillageReach’s case for impact and ability to use additional funds to have additional impact is significantly more compelling to us than that of any other organization we’ve found. If Holden had a similarly strong option to help, say, Indian youth, he would likely opt for that, but as it stands, VillageReach continues to be the best option we know of for helping any population.

We will continue to search for strong options for donors who most want to help different populations. In the meantime, we’d love to hear your thoughts on this debate in the comments section below.

A few disclosures regarding this book:

• Innovations for Poverty Action is a client of Sona Partners, and Tim Ogden of Sona Partners sits on our Board.
• We are currently considering Innovations for Poverty Action for a recommendation.
• We sent the authors of the book an early draft of this review and have made modifications after some back-and-forth.

I recently read More than Good Intentions, a book by Dean Karlan and Jacob Appel of Innovations for Poverty Action (which we’ve written positive things about before).

• I thought the book was excellent, and I highly recommend it to anyone interested in effective giving and in development economics. I especially recommend it to anyone interested in microfinance – it has the best extended discussion of microfinance that I’ve seen.
• I am a bit concerned that the book doesn’t adequately capture the distinction between “exciting ideas with some preliminary evidence behind them” and “the most proven interventions in aid.” I feel that programs falling in the latter category are largely not discussed in the book, and that this fact may be unclear to donors unfamiliar with the area.
• The closing chapter of the book points readers to the Proven Impact Fund, which we understand is still a work in progress. After some discussion with Innovations for Poverty Action, we have determined that the Proven Impact Fund is unlikely to receive a GiveWell recommendation until and unless it becomes more established and has a track record. (We will be publishing the details of our views on the Proven Impact Fund at another time.) We are, however, evaluating Innovations for Poverty Action itself as a vehicle for donors to contribute to more high-quality research on aid.

Excellent discussion of microfinance and other topics

The book opens with a multi-chapter discussion of microfinance. Dean Karlan has been a leader in high-quality studies of microfinance, many of which have debunked the myths surrounding it, and I find his discussion of it to be the best I know of: engaging, educational, and sensible in its interpretation of the information we have.

The discussion centers around the idea – often lost in the stylized stories around microfinance – that different people have different needs, and that loans can be helpful to some and harmful to others. Karlan and Zinman’s study on consumer loans in the Philippines makes a strong case that when the timing is right, a loan can be very helpful even at a high interest rate. But the book also discusses people like Erlyn, whose business had a limited need for capital, such that the large long-term loans associated with her microfinance institution largely served to put her deeper into debt. The work of Karlan and others has shown how harmful many of the restrictions and rules of traditional microfinance – often based on romanticized notions of poor entrepreneurs – can be.

After discussing microfinance, More than Good Intentions goes through research on how to help the poor in other ways, from improving access to health services to raising attendance in schools. I was familiar with most of these studies prior to reading the book, but the book adds helpful context on the background and motivation for the various research projects.

One of the consistent themes of the research is the tradeoff between (a) giving clients the freedom and power to make their own choices; (b) finding “nudges” that can help people act more strongly in their self-interests. Much of the microlending research suggests that more freedom and discretion is better, but in other areas, clients seem to benefit from some restrictions – and in some cases the restrictions are even self-imposed (as with the SEED program, in which clients open savings accounts that can’t be withdrawn from until they save a certain amount).

It does seem to me that the set of investigations discussed in the book is missing an important piece. The studies cited generally compare one version of a charity’s program to another, or they compare it to “no intervention”, rather than compare a program to simply handing out cash equivalent to the program’s costs. To us this seems the most appropriate way to ethically and reasonably evaluate the value of programs: put them up against the simplest, most direct, and most empowering/discretionary of all interventions, a cash gift.

We understand that there have been studies in this field that do investigate unconditional cash transfers vs. other interventions; we also understand that funders are not always willing to finance a comparison to cash transfers as opposed to “no treatment.” Still, the fact that microfinance programs, free school uniform programs, etc. have always been compared to “no treatment” rather than “cash” seems suboptimal to us.

I also want to highlight the following anecdote from the beginning of the book, as a reminder of what low-quality evaluation (even at the largest and best-known organizations) can look like:

The first “impact evaluation” I ever saw of microcredit made my stomach ache. It was clearly intended to generate pretty numbers for a brochure to donors – not to determine whether something was really working. It asked clients something like, ‘You are eating better now, compared to before you joined FINCA, yes?’ … What I have since learned is that FINCA was doing just as much to measure its impact as anyone else. Which is to say, very little.

“Proven” vs. “exciting”

The book overall covers a lot of ground but doesn’t cover all the ground we would have liked it to, given its ambitions to encourage people to give to proven, effective interventions. This is best illustrated by the final section of the book, which aims to “leave you with seven ideas that have me [authors] particularly excited.”

The seven ideas are microsavings, reminders to save, prepaid fertilizer sales, deworming, remedial education in small groups, chlorine dispensers for clean water, and commitment devices. (The list of highlighted ideas has since been updated, as noted below.) I feel very strongly that this list (and its update) do not include the programs with the best track records in international aid. In my view, several proven health interventions – most notably immunization campaigns, distribution of insecticide-treated nets, and tuberculosis case finding and treatment – all have far better and more robust track records than any of the seven. The reasons I feel this way have to do with the distinction between micro evidence and macro evidence: the ideas in More than Good Intentions are all primarily supported by micro evidence (programs that took place on a small scale and were monitored and evaluated by academics, with the ultimate proof of effectiveness consisting in numerical differences between treatment and control groups) whereas the ideas I have listed are supported by both (not just randomized controlled trials, but also country-level success stories where a scaled-up program achieved an impact that is easy to see outside the confines of a study). I may elaborate on why this distinction is important to me in a future post.

I’m not sure that the authors of More than Good Intentions would disagree. So why does the book focus on relatively new ideas and give less attention to programs such as traditional immunization campaigns? I think there is a perfectly reasonable explanation:the authors are particularly excited about interventions that they and their colleagues have helped prove the promise of, as opposed to interventions like immunization campaigns that have been recognized as effective for decades.

There’s nothing wrong with being particularly excited about insights that are particularly new, counterintuitive, or simply a source of personal pride. However, donors need to recognize the difference between “exciting new ideas worth learning more about” and “the programs with the best track records,” and I am not sure that they will come away from More than Good Intentions with this distinction clear to them.

The closing chapter of the book points readers to the Proven Impact Fund, which we understand is still a work in progress (and unlike the list of ideas in the book, it includes interventions related to bednets and immunizations – though not the same interventions that I feel are most strongly associated with macro-level successes). After some discussion with Innovations for Poverty Action, we have determined that the Proven Impact Fund is unlikely to receive a GiveWell recommendation until and unless it becomes more established and has a track record. (We will be publishing the details of our views on the Proven Impact Fund at another time.) We are, however, evaluating Innovations for Poverty Action itself as a vehicle for donors to contribute to more high-quality research on aid.

In the meantime, we highly recommend More than Good Intentions as one of the most sophisticated discussions we’ve seen of microfinance and other fronts in the fight against poverty.

There has been a lot of coverage of the scandals around the Central Asia Institute. The founder has been accused of fabricating inspiring stories, as well as of spending less than half of the millions of dollars he’s raised on building schools.

The Central Asia Institute receives four stars from Charity Navigator (archived) and had perfect ratings from Great Nonprofits prior to the scandal (archived), but GiveWell has declined to give this organization a recommendation or distinction (note that that page was published in mid-2009).

In a sense this doesn’t indicate impressive foresight on our part: nearly all charities we examine do not receive recommendations or distinctions, so it’s not as though we spotted the fabrications and financial mismanagement ourselves. Yet in the bigger picture, I see this incident as a vindication of our approach to giving: it’s a reminder that you shouldn’t give charities the benefit of the doubt.

Central Asia Institute is an education charity. Our questions for education charities are:

• Is the goal to improve attendance or to improve school quality?
  • If the goal is to improve school attendance, what sorts of schools will beneficiaries be attending? What information is available on teacher attendance and quality of instruction? What evidence is available regarding the program’s effect on attendance?
  • If the goal is to improve school quality, is there evidence that similar activities have causally led to improved schools (in terms of attendance, test scores, graduation rates) in the past? Are indicators of school performance tracked over time, both before and after the interventions?
• What evidence is available regarding the likely impact of any improved performance/attendance on later life outcomes? (For example – what are the economic opportunities that will be available to students and how do education levels relate to them?)
• How would activities change if more revenue than expected was received? Would more revenue translate into more students served, and up to what point?

When we looked at Central Asia Institute’s website, we saw lots of stories but not a word to answer any of these questions substantively. So, as we usually do, we declined to give it a recommendation or distinction. As it turns out, the Central Asia Institute couldn’t have answered these questions well, because (a) it does appear to focus on school building rather than school quality, thus falling prey to one of our main concerns about education charities; (b) it apparently does not have strong monitoring and evaluation of its own schools. From the CBS story:

The IRS tax return Central Asia Institute filed last year included a list of 141 schools that it claimed to have built or supported in Pakistan and Afghanistan. Over the past six months, we visited or looked into nearly 30 of them. Some were performing well, but roughly half were empty, built by somebody else, or not receiving support at all. Some were being used to store spinach, or hay for livestock; others had not received any money from Mortenson’s charity in years.

If the Central Asia Institute had had anything on its website to indicate promise (such as an example study from a single school showing high-quality education), we would have opened a more in-depth investigation, including in-depth examination of the “What do they do?” and “Is there room for more funding?” questions, which would certainly have led us to the observation that the Central Asia Institute spends less than half its money on actually building and maintaining schools. (This point, raised by the CBS story, is easy to see from page 2 of the organization’s most recent tax return (PDF).) The fact that it still technically has a low “administrative expense ratio” is just a reminder of how easily manipulated, and ultimately meaningless, this metric is.

People frequently object to the fact that we recommend so few charities, and to our equating of “not enough evidence to evaluate” with “not recommended.” But the fact is that if a charity isn’t disclosing substantive information to answer the tough questions about its activities, you have no way of knowing whether it has the same kind of problems that the Central Asia Institute has. Certainly you can’t know from looking at its rating from other agencies.

We recommend very few charities, but we are able to stand behind those we do recommend. If you’re looking to do as much good as possible with your donation, consider choosing from a smaller menu to ensure that you get quality.

We recently published an updated review of PSI based on conversations over the past year with PSI staff and additional analysis. We now rate PSI as a “Notable” organization, and no longer list it among our top-rated charities. (Note that PSI is not the only organization whose rating changed in 2010: we similarly “downgraded” both Partners in Health and the Global Fund to “Notable” and the Stop TB Partnership to “Silver” from “Gold.”)

PSI is the organization GiveWell has followed the longest:

• I chose PSI as the recipient of my first substantial charitable donation in December 2006 (the outcome of my GiveWell research when GiveWell was just a part-time project).
• In GiveWell’s first round of full-time research in the fall of 2007, PSI was our top-rated charity in our “Saving Lives in Africa” cause, and GiveWell sent PSI a $25,000 grant. We believed that PSI’s cost-effectiveness was roughly in-line with its own $/DALY estimates.
• In mid-2009, we conducted a more thorough and in-depth analysis of international aid charities reviewing hundreds of organizations (instead of merely reviewing the ones that applied for a grant as we had in 2007). In that process, PSI maintained its “Recommended” status and we ranked it fourth out of the hundreds of charities we had considered. In that report’s review of PSI’s impact, we questioned whether PSI’s programs cause behavior change (such as condom use by individuals who did not previously use condoms).

What changed?

In 2010, we returned to all our previously completed charity reviews to reconsider them and update them with current information. In that process, our conclusion about PSI changed, due specifically, to (a) our view of the way in which PSI uses and evaluates the monitoring data that they collect and (b) new data we found that provided estimates for changes in health behaviors across all of Sub-Saharan Africa.

In our July 2009 review of PSI, we had concluded that

Data from [PSI’s monitoring and evaluation] surveys indicates that PSI’s products are likely being used (more below). It is unclear, however, whether the availability of PSI’s products causes behavioral change (such as condom use by individuals who did not previously use condoms).

While we weren’t sure that PSI’s programs were effectively creating behavior change, we felt that PSI was carefully evaluating its programs in a way that would identify failing programs and lead to improvements. We were ambivalent about PSI’s impact but felt good enough about its monitoring and evaluation process to recommend it. We believe that a charity (a) focused on programs with strong track records (as PSI is by promoting condom and bednet use) that (b) has an unusual degree of transparency while (c) carefully evaluating its programs to determine whether they are working is worth recommending to donors.

Earlier this year, PSI questioned our conclusion that its programs don’t cause behavior change. PSI made its case for impact by stating that “out of 163 available multi-round studies of behavior change, 82 have shown statistically significant impact on at least one behavior change indicator.” We didn’t find this case compelling as we see many ways PSI could measure these results without having significant impact. Specifically:

• PSI’s test for “impact” – a single indicator showing a statistically significant change in behavior – seems unreasonably generous. PSI didn’t offer a compelling justification for the test it used. This led us to question the quality of process PSI uses to monitor and evaluate whether its programs are largely succeeding or failing.
• We also found data providing estimates for bednet use across Sub-Saharan Africa and compared this data to measured changes in PSI’s focus areas to broader changes in bednet and condom use across Sub-Saharan Africa. Our analysis of this data shows that changes of health product use in PSI programs seems in line with changes in non-PSI areas across Sub-Saharan Africa (details in our review of PSI’s impact): PSI’s average change was 3-4% for condoms (compared to 2-3% across Sub-Saharan Africa) and 12-14% for bednets (compared to 23-28% across Sub-Saharan Africa). While we do not believe this demonstrates that PSI is not having impact, this data does shift our assessment of the likelihood PSI’s programs are effective and increases the burden of proof on PSI to demonstrate its programs’ effectiveness.

In part, the downgrade is a reflection of the fact that we are continually digging deeper and raising the bar higher for our recommendations. When we first encountered PSI, it was the only nonprofit we could find that seemed to have any meaningful data on its impact; the limited slice of data we looked at seemed to indicate impact and to be consistent with its own cost-effectiveness estimates; and that was enough for us. By now, we are more careful about demanding data that is broadly representative of the full organization’s activities, and in our fuller analysis of PSI’s data we were less compelled. We also place high value on admitting and responding to shortcomings. If PSI agreed with our view that its data does not convincingly demonstrate impact, we would consider a higher rating based on discussions of its ongoing learning and adjustment.

We still believe that PSI stands out from the vast majority of international aid charities because of its focus on programs with strong track record and its unusual and commendable degree of transparency. (If I had to guess I’d say that PSI is probably in the 99th percentile of all international aid charities in terms of its impact per dollar.)

Nevertheless, our focus at GiveWell is identifying the very best charities and for the reasons above, we no longer feel comfortable placing PSI in that group. We plan to continue to watch PSI carefully and reevaluate its status as new information becomes available.