The GiveWell Blog

Development experiments (randomized controlled trials) as a counterpoint to marketing materials

(updated on: ) | by Holden

There’s been a minor flurry of recent blog posts about randomized controlled trials (RCTs) in international aid, including William Easterly’s take and responses by Chris Blattman and GiveWell Board member Tim Ogden.

A central theme has been the difficulty of generalizing from one experiment to whether something “works in general.” There seems to be a loose consensus that while RCTs can highlight interesting behavior patterns, they shouldn’t be expected to stamp programs with “effective” or “ineffective” to the extent that a page like this one suggests.

To me, this concern is another example of “two worlds” in philanthropy. Scholars naturally consider it an open question whether a program “works,” and rightfully point out that a single experiment (or even many experiments) can’t provide a general answer; within this context, the cautions about RCTs are good to note. But charities constantly “sell” the logic of a single approach, independent of time and place; within this context, RCTs provide a meaningful and much-needed reality check.

Microfinance is an excellent example. The Grameen Foundation states, “Microfinance helps people to escape poverty by giving them collateral-free loans and other financial services to support income-generating businesses.” Though it works in four continents, its success stories all follow the same basic pattern, in which a loan is used to expand a small business. Messaging from other microfinance organizations is generally similar, leading to what I perceive as a highly unified impression among donors that microfinance fights poverty by enabling small business expansion.

In this context, the argument of a recent paper (PDF) is highly relevant. It asserts (and we have reviewed it only superficially at this point) that, among other things,

  • The impact of microloans in this case (an expansion of credit in Manila) was generally to make businesses shrink (“treated businesses shed unproductive employees. One explanation is that increased access to credit reduces the need for favor-trading within family or community networks” – pg 3)
  • Loans were more beneficial for “male, and relatively high-income, borrowers” (pg 4)

These claims substantially challenge (a) the story that microfinance institutions have been using to explain and “sell” their operations; (b) the way they have been carrying these operations out; and (c) the assumptions that donors normally bring and the questions they normally ask.

From what we’ve seen, charities don’t proceed with the caution of researchers, trying to fully understand what happened in one area before moving on to the next. They tend to run with a single theory of how an intervention changes people’s lives, scaling up the intervention substantially – often in many countries – with little or no evaluation.

Hence the value of even a single truly rigorous test, in a single time and place, of whether reality was consistent with the theory. We recognize the limitations of RCTs (and have not relied on them much in generating our current recommendations), but we feel strongly that a charity is far more credible when it is testing its claims with RCTs, and that the existence of more RCTs would greatly help donors.

Can donors fund “sustainable” projects?

(updated on: ) | by Holden

It’s one thing to pay for children’s immunizations. It would be quite another to pay for a project that increased immunization rates over the long term, without continued donor support (either thanks to improvement in private-sector or government operations).

Aiming for the latter – or more broadly, aiming to use donations as “startup funds” for lasting and self-sustaining improvements (rather than as indefinite subsidies) – is often referred to as “sustainable development.” Pursuing “sustainability” is often very important to funders. (For example, the World Bank’s latest Independent Evaluation Group report aggregates figures on the likelihood of sustainability for all projects evaluated, and lists these aggregates as one of its key indicators – see Table A.4).

Yet while we would far prefer sustainable programs to unsustainable programs, we don’t currently list “sustainability” as one of our key criteria.

The reason is that while we’ve seen many charities aiming for sustainability, we’ve yet to see any that can demonstrate both future plans and past success in this area.

Sustainability is hard to come by

Most of our recommended international aid charities are aiming at least partially for lasting impact, but evidence that they’ve achieved it is murky at best. Two examples:

Non-recommended organizations rarely appear to be documenting long-term impact of any kind.

Independent literature reinforces the concern that achieving sustainability is difficult and rare.

  • A 2004 paper entitled “The Illusion of Sustainability” states, “Some anecdotal evidence suggests that sustainability has often been a chimera – and sometimes a costly one.”
  • A 2008 paper by William Easterly adds, “This hope [for sustainable projects] has turned out to be an illusion, as the failure to cover recurrent costs has been nearly universal.”
  • A book reviewing aid effectiveness (and generally positive on it) states that “The bulk of the NGO evaluation material which looks into the issue of sustainability has concluded that most NGO projects are not financially sustainable without the continued injection of external funds … Exceptionally few studies have provided a long-term view of [nonprofit] projects and programmes. Most views on the sustainability of NGO projects are based on forward extrapolation of short-term assessments” (281).

We do believe that sustainability in aid is possible, and has been achieved in the past. For example, Aravind Eye Care System appears to be a self-supporting humanitarian organization, using revenue from paying customers to perform cataract surgery on those with lower incomes. This program, in fact, appears so sustainable that there is little donors can add to it, and donations support tangentially related activities. (More broadly, we were recently pointed to family planning as an example of an area where aid has been associated with lasting behavior change.)

But as with impact, we believe that sustainability needs to be demonstrated, not just claimed. In fact, the burden of proof for demonstrating sustainability is even higher than for demonstrating impact.

The importance of long-term evaluation

We constantly stress the importance of systematic, long-term impact evaluation, and note how rarely it is seen in the world of charity. It seems to us that if one is pursuing sustainable impact, the challenge and importance of evaluation become more significant. In order to assess sustainability, a charity must be systematically examining an area (with all the challenges that involves) even after it has withdrawn support.

VillageReach appears to be doing exactly this, and is therefore at least in a position to identify and respond to disappointing results (according to VillageReach representatives, the local government appears to have lapsed back into its former, less efficient logistics approach, but has agreed to renew its focus on sustaining the new model).

In our view, any project that does not include rigorous evaluation can’t make any credible claim that it has achieved sustainability – only that it hopes to.

A desirable goal, but not a reasonable requirement

We think it may often be wise for experts and major funders to focus on sustainability – doing so could make sense even with a high failure rate, because successes are so significant.

But in trying to help individual donors – who aren’t well positioned to innovate – we focus on finding programs that have worked before and are likely to work again. If we required recommended charities to show evidence of “sustainability,” we simply would have no recommended charities, at least at this point. As such, we’ve evaluated our strongest charities by what they can accomplish even assuming that their hopes of sustainability don’t pan out. We recommend VillageReach, despite concerns about sustainability, because its program appears to do a lot of good for relatively little money even if its improvements are never maintained (more here).

If VillageReach’s program does end up having a lasting impact, it will be an even better deal. If we later find a charity that can convincingly demonstrate an ability (not just a plan) to create sustainable and demonstrable impact, we will recommend it strongly. But in the meantime, it doesn’t seem reasonable to criticize a charity on the grounds that its work isn’t sustainable. Simply having real, demonstrable impact is hard enough.

2008-2009 international aid recommendations

(updated on: ) | by Holden

Our 2008-2009 recommendations for international aid are now available.

Highlights:

Much more discussion to come in the following weeks.

A conversation with a donor interested in obstetric fistula

(updated on: ) | by Elie

Jeremy, a teacher, recently emailed us because he was interested in supporting charities that work on the problem of obstetric fistula. Fistula is a cause that I’ve written about before and is one that, emotionally, is extremely compelling.

I thought the email exchange with Jeremy was informative, so (with his permission) we’ve posted it here.

In brief, I think it provides:

  1. A reasonable template for a donor thinking about supporting an organization that provides surgical care
  2. An example of how a donor can care deeply about a specific cause and have an initial feeling about the best organization to give to but then benefit from thinking critically about his or her giving decision.

Donating to Gates – against its will

(updated on: ) | by Holden

With all the people and organizations out there who would like more money, there’s something remarkable about the fact that the Gates Foundation is specifically asking people NOT to give to it (PDF) – and still couldn’t keep away over $10 million in donations in 2008.

Why the Gates Foundation doesn’t want your money

First off, I have a lot of trouble understanding the Foundation’s stated reasoning (PDF):

From time to time, people generously offer to contribute money to the foundation. We prefer that people give directly to our grantee organizations rather than to the foundation if they want to help advance the causes we’re passionate about. We have the stable funds we need to help us fulfill our mission, but our grantees often do not.

It seems to me that there are a couple of problems here.

  • Is it desirable that the Gates Foundation’s grantees should have more “stable funds” than they do now? If so, why doesn’t the Gates Foundation give them these “stable funds” in the form of unrestricted grants? Couldn’t it, at the very least, use its “extra money” (the money that comes in as individual donations) in this way?

    How does it make sense for the Foundation to ask people to fund something that that they choose not to fund themselves? More broadly, how does it make sense for an organization to that exists for the sole purpose of giving away money as well as possible to be discouraging people from giving to it?

  • The Foundation directs people to its list of grantees – in the form of a database of (currently) over 5,000 grants. How is an individual donor to cut through this information and figure out the best fit?

Speaking as someone who personally tried to do exactly this about 3 years ago, I can tell you that the information the Gates Foundation is providing is nowhere near sufficient to figure out which nonprofits would most benefit from my donation. I’m honestly surprised that the Gates Foundation is pointing people to pages like this one (and the other pages it links to) for information. The Gates Foundation has a large and well-credentialed staff devoted to researching where to give out money, yet they’re asking individuals to make the decisions themselves based on profiles that read like fundraising brochures?

The bottom line is that I don’t think the Gates Foundation makes a compelling argument that you shouldn’t give to it (so perhaps it’s not surprising that so many people have chosen to). To me, the strongest part of its case is the implication that it actually considers pages like this one to provide all the information and analysis a donor needs. If that were the case (and I doubt that it is), I’d definitely prefer to use my own judgment.

What does $10 million mean?

Sean hypothesizes that “this is direct evidence of individual donors’ increasing interest in impact.” I’m inclined to agree.

We often hear that donors don’t really care about having the most impact possible; they care about attending benefits, or dispelling the guilt/cognitive dissonance raised by an appeal, or identifying with a cause. We respond that at least some donors are motivated primarily by wanting to make the world a better place; the question is how many of these are out there (because we know that the total pie is quite large). And we honestly don’t know the answer to that question – better research is needed.

But now we know that 2008 saw at least $10 million come from people who could not have been motivated by social events (the Gates Foundation holds none and doesn’t allow fundraisers on its behalf), in-the-moment emotions (the Gates Foundation conducts no appeals and doesn’t have a particularly cute namesake), or identification with a pet cause (the Gates Foundation’s work includes U.S. education, international aid and more, and it doesn’t allow people to earmark for a specific cause).

The only reasoning I can think of for giving to the Gates Foundation is, “They’ll do a better job with this [in terms of making the world a better place] than I can, and that’s what I want even if it doesn’t come with the donor perks of a traditional charity.” If you disagree and can think of a less altruism-based reason, please share in the comments.

Of course, that $10 million could all have come from 1-2 people for all we know (especially since only $1.6 million came in the year before). More on this below.

Will the Gates Foundation help us learn more about these donors?

Correction (added 6/11/09): Sean Stannard-Stockton has pointed out that the Gates Foundation will be releasing the names and amounts for all donors who gave more than $5000, as required by law. Assuming this is correct, the remainder of this post (from here until the end) should be disregarded.

There is growing interest improving donors’ access to quality information. In order to do this well, it would help greatly to know whom to target – in particular, whether the lion’s share of “impact-focused” charity is coming from tiny, medium, large, or mega donors. At this point we know so little about this question that having the breakdown of donation sizes from the Gates Foundation would, I believe, add a lot to our understanding.

I believe the Gates Foundation could be a good citizen and helpful to the cause of improving philanthropy by releasing information on how many donations it received of different sizes. Aggregating them by “buckets” (<$100, $100-500, etc.) and keeping all names confidential would allow this information sharing without compromising anyone's privacy. Will they do it?

The challenge of local ownership

(updated on: ) | by Holden

One of the consistent refrains we’ve seen in aid literature is the importance of local participation/enthusiasm/ownership for aid projects. Many programs have been criticized for being too “top-down” (i.e., imposing outsiders’ designs on local communities), with the implication that more “bottom-up” programs (i.e., getting local people to participate in the design of execution of programs) would be more likely toi create real and lasting change. For an example of this reasoning, see this USAID review of Integrated Rural Development programs (PDF).

The basic reasoning makes sense, but making a program “bottom-up” is easier said than done. For an illustration of why, see this World Bank review of “community-based development,” a term referring to “projects that actively include beneficiaries in their design and management.”

The frequent tendency for participatory projects to be dominated if not captured by local elites is highlighted by several case studies. Katz and Sara (1997), in a global review of water projects, find numerous cases of project benefits being appropriated by community leaders and little attempt to include households at any stage … even well trained staff are not always effective in overcoming entrenched norms of exclusion. In a study of community forestry projects in India and Nepal that worked reasonably well, Agarwal (2001) reports that women were systematically excluded from the participatory process because of their weak bargaining power. Rao and Ibanez (2003) find that in the participatory projects in their Jamaican case study, wealthier and better networked individuals dominated decision making. In a similar case-based evaluation of social funds in Jamaica, Malawi, Nicaragua, and Zambia, the World Bank (2002) Operations Evaluation Department concludes that the process was dominated by “prime movers.”

Abraham and Platteau (2004) present evidence on community participation processes in Sub-Saharan Africa based largely on anecdotal evidence from their work in community-based development and on secondary sources. They argue that rural African communities are often dominated by dictatorial leaders who can shape the participation process to benefit themselves because of the poor flow of information. (40-41)

These notes capture a concern of ours that applies to all aid projects: while the goal is to help those in the most need, those with the least need may be most likely to have the resources, connections and free time to get the inside track on any particularly generous aid project. This is also a major reason to be skeptical of simple evaluations comparing “project participants” to “non-participants,” as many microfinance evaluations do. Project participants may simply be better off to begin with (and some studies show that they are, such as the Coleman study referenced on the previous link).

We don’t believe that a simple and straightforward way to overcome this challenge is available. That’s why, although we agree with the basic concept that local ownership will improve a project, we don’t tend to judge projects by their formal commitment to local ownership – i.e., we don’t favor programs that work in formal community votes, meetings, etc. over programs that don’t. The former could be improving local participation or transferring more power to elites; the latter could be generating local enthusiasm simply through a good match between what people want and what they’re being offered.

It’s easy to claim that one is involving community members, but the ultimate test is in outcomes – whether the project ran well enough and generated enough local participation to accomplish its ultimate goals (improved health, incomes, etc.)