In a recent debate, David Hunter’s article on the nonprofit sector has taken heat for its assertion that “While nonprofits work incredibly hard, with passion and dedication, and often in incredibly difficult circumstances to solve society’s most intractable problems, there is virtually no credible evidence that most nonprofit organizations actually produce any social value.”

We agree with the claim for the sectors we’ve examined, which we believe are similar to the sectors Mr. Hunter has examined: particularly thorny areas such as charities working to improve education and international charities addressing extreme poverty overseas. These are problems on which experts have struggled for decades to make any progress, and while we don’t necessarily agree that most charities are failing to produce value, we agree that most charities cannot produce any credible evidence that they are. This is different from the claim that Sean Stannard-Stockton attributes to Mr. Hunter (”most nonprofits and the social sector as a whole is not currently producing social value”), but it still means that it’s very hard for a donor to give with confidence.

The information we have

Our belief is based on two years of looking for this evidence; we’ve published the full details of our findings online, and you can see our summary of international charities (only 19 out of 320 examined publish any impact-related evaluation reports) and U.S. equality of opportunity charities (only 6 of 83 examined provide credible impact-related reports, and 2 of these show negative or no impact).

In addition, in a guest post on the GiveWell Blog, David Anderson of the Coalition for Evidence-Based Policy estimates that 75% of rigorous evaluations show weak effects, no effects, or negative effects.

More information needed

On the other hand, we also believe the criticism that Mr. Hunter doesn’t support his own claim with evidence has merit. We would like more clarity on which sectors Mr. Hunter has examined and is referring to, and information on where he has looked for evidence and what he has found.

In addition, we feel that examples of failing/harmful programs, such as “Well intentioned but ineptly run mentoring programs where failed matches reinforce in youngsters a sense of their low worth and poor prospects” (and the other items on the list on page 2 of the article) should be clearly referenced to summaries of evidence.

The truth is that we cannot have a very informed debate about how much value nonprofits create because we have so little evidence of any kind. Some people adamantly believe that nonprofits create enormous value; others are skeptical that they create any; and there is very little to go on, at least in the sectors under discussion.

Nonprofits that do have credible evidence of their social impact

The good news for donors is that they need not be in the dark if they give to the right charity. Our top-rated charities do produce credible evidence of their social impact. We encourage individual donors to expand and fund these charities until and unless others follow suit.

GiveWell generally focuses on the question of how to get “bang for your buck” as a donor - help as many people as possible, as much as possible. Against this approach, one might seek to factor in the potential of a program to get at the “root causes” of poverty, and start - or be part of - a chain reaction that ends poverty at the country or even world level. (One example of such reasoning in nonprofit marketing is here.)

Below is our take on the following broad question:

Why have some parts of the world emerged from poverty while others haven’t? How can financial aid from developed nations best be directed to cause large-scale emergence from poverty?

To us the key points are:

Past emergences from poverty have taken many different forms; there are no clear/consensus patterns or formulas in these stories.

This statement - which we have seen little to no literature contradicting - is well illustrated by the work of two major commissions, each of which set out to find patterns in the history of economic development.

A 1993 World Bank study on the rapid growth of 8 countries in East Asia concluded (overview; similar content on page 366):

The study attempts to explain East Asia’s success and to develop a model of rapid growth with equity. It finds that the diversity of experience, the variety of institutions, and the variations in policies among the [high-performing Asian economies] does not allow a model to be developed.

A more recent (2008) study, “led by 19 experienced policymakers and two Nobel prize-winning economists” (from the press release), contains similar caveats (from page 2):

The report identifies some of the distinctive characteristics of high-growth economies and asks how other developing countries can emulate them. It does not provide a formula for policy makers to apply—no generic formula exists. Each country has specific characteristics and historical experiences that must be reflected in its growth strategy. But the report does offer a framework that should help policy makers create a growth strategy of their own. It will not give them a full set of answers, but it should at least help them ask the right questions. Fast, sustained growth does not happen spontaneously. It requires a long-term commitment by a country’s political leaders, a commitment pursued with patience, perseverance, and pragmatism.

There are many different prescriptions for the actions most likely to end poverty.

Various scholars propose large-scale plans - based on theories of the root causes of poverty - for ending (or drastically reducing) poverty. However, it should be noted that their plans (a) are very different from each other; (b) tend to be highly multidimensional and to depend heavily on governments and/or international institutions. (In other words, few put a single “silver bullet” intervention at the heart of their plans.) Three prominent examples:

• Jeffrey Sachs (The End of Poverty) argues that poverty itself is self-reinforcing: when people and governments have low enough income, they cannot make the necessary investments to create strong future growth (see pages 245-250). His recommendations, presented in the UN publication Investing in Development, center around aid to developnig-world governments, in support of multidimensional and country-specific poverty reduction strategies (page xx). Such aid is to be roughly doubled from 2003 levels by 2015 (page xxii); other recommendations include loosening trade restrictions and increasing relevant scientific research (page xxii).
• Paul Collier The Bottom Billion) believes the world’s poorest countries are caught in one or more of four “traps” (page 17): patterns of civil war (chapter 2), “resource curses” in which large natural resource wealth prevents healthy economic development (page 39), the condition of being “landlocked with bad neighbors” (53), and bad governance in small countries (65). His agenda (177-183), like Sachs’s, includes an emphasis on free trade, but it also includes major roles for military intervention to deal with conflict as well as international laws and charters to hold governments accountable. Although he supports aid to some governments (179), he specifically cautions against the dangers of too much aid in some cases (page 181), and proposes circumventing governments through “independent service authorities” in others (177, 179).
• Peter Timmer (Agriculture and Pro-Poor Growth: an Asian Perspective) argues that “No country has been able to sustain a rapid transition out of poverty without raising productivity in its agricultural sector” (page 3) and recommends a set of interventions focusing on rural areas, particularly agriculture (pages 29-30).

There are reasons to see health aid as a promising approach to reducing poverty. There are also reasons to see it as neither necessary nor sufficient for this goal.

The idea that improving health would lead to improved productivity, and thus less poverty, is one that makes intuitive sense and has some suggestive evidence to support it. Working Group 1 of the WHO Commission on Macroeconomics and Health (PDF) summarizes many different kinds of evidence including cross-country analysis, studies on the productivity of people receiving nutrition supplements, and analysis of changes in calories available over time (see pages 5-12 for an overview, although we are looking for a clearer and more complete review on this topic).

But there is little reliable guide to how much economic improvement can be expected to come directly from health improvement, and even strong advocates of health aid such as the Commission on Macroeconomics and Health do not see health aid as sufficient (by itself) to end poverty. (See pages 28-29 of the commission’s final report.) There is also no consensus that health aid is necessary; some see past emergences as having been led by agricultural improvements (such as Timmer, discussed above) and/or having been accomplished largely without external assistance of any kind (as William Easterly does - see page 347 of White Man’s Burden).

Bottom line

No single theory of the “root causes of poverty” is supported by overwhelming evidence or broad expert consensus. We don’t find any compelling enough, or relevant enough to what individual donors can do, to compete strongly with the goal of improving individuals’ lives - a goal that could itself be the best approach to speeding the end of poverty, particularly if you believe (as scholars such as William Easterly do) that the emergence of nations is most likely to be homegrown.

There are many proven, cost-effective, scalable ways to significantly improve people’s lives. We feel that adding to one is the best use of an individual donor’s funds.

While Elie’s been investigating the Carter Center, I’ve been scanning literature (mostly academic) on general questions about aid: what has worked in the past? What’s promising for the future? etc.

Rather than trying to come to an independent conclusion on each debate, I’ve been trying to establish which beliefs are supported by evidence that is largely undisputed among scholars (and where there is no agreement, get a sense of what each side believes and what its most frequently pointed to evidence is).

At this point I’ve looked in a lot of places (though I’m still far from done) and I’m going to start sharing where we stand (and what we’re still missing) on various questions. First I’ll discuss what I’ve read about the track record of aid to developing-world areas.

The vast majority of the large-scale “success stories” I’ve seen come from health initiatives (particularly “vertical” health initiatives, i.e., large-scale campaigns against particular diseases).

• Millions Saved: Proven Successes in Global Health (published by the Center for Global Development) seems to be the most frequently cited reference by authors seeking to point to “success stories.” It lists 20 large-scale case studies including the global eradication of smallpox and the Carter Center-led campaign against guinea worm.
• There is also a success stories compilation by the World Health Organization, though these case studies don’t provide data sources (and haven’t responded to our request for them) and in many cases seem to be discussing smaller-scale initiatives.
• Jeffrey Sachs’s The End of Poverty lists ten “significant examples of programs that have been scaled up massively to remarkable success” (pgs 260-265). They are again dominated by health interventions. My notes on them are available here (via our public email list).

The main non-health “success stories” I’m aware of:

• Philanthropic funding of agricultural research is often credited with a major role in catalyzing the emergence from poverty of many countries in Asia and elsewhere (an emergence often referred to as the “Green Revolution” because of the prominent role of productivity gains in agriculture). So far, efforts to expand the “Green Revolution” to Africa have largely been disappointing (overview on pgs 66-72 of Can the West Save Africa? (PDF) by William Easterly).
• There is “suggestive” evidence that education aid has resulted in increasing primary enrollments, though the effect of this increasing enrollment on growth/standard of living/etc. is unclear (overview on pgs 47-53 of Can the West Save Africa? (PDF) by William Easterly).

I don’t believe that any of the sources cited above are fully comprehensive (or necessarily fully reliable) as lists of large-scale success stories. We’re still looking for more.

Aid has also had major failures - but insufficient monitoring and evaluation means that few are thoroughly documented or discussed.

• Dissatisfaction with the accomplishments of aid to date is fairly widespread. The latest report on the Millennium Development Goals shows mostly inadequate/less-than-hoped-for progress (and no progress in many cases). William Easterly argues that
  

  systematic testing would not just count the alleged ’success stories’ of aid, but also the larger number that got the same amount of aid as the ’success stories’ and failed: Guinea-Bissau, Somalia, The Gambia, Mali, Rwanda, Nicaragua, Burundi, Guyana, Zambia, the Central African Republic, Senegal, Suriname, Chad, Niger, Togo, Haiti, and so on. Further testing shows that these outcomes were not an artifact of selection bias or reverse causality.

• Yet while project-level evaluations provide scattered analysis of projects gone wrong (one example of a failed World Bank project here), we have not found well-documented “failure stories” along the lines of the “success stories” above - examining major humanitarian (as opposed to political) initiatives that had little or negative impact.
• Part of the reason may be the general lack of evaluation and documentation in international aid. The sentiments of this Center for Global Development paper, arguing that “very few programs benefit from studies that could determine whether or not they actually made a difference. This absence of evidence is an urgent problem,” are common; the Paris Declaration on Aid Effectiveness, an agreement between major aid agencies, includes an expressed commitment to more monitoring and evaluation.

I have the sense that failure stories in aid are common, but I continue to look for more concrete examples of what has gone wrong and how.

The relationship between aid and growth at the macro/country level has been extensively studied, but is not well established one way or the other.

Some believe in a moderate positive relationship, often with the caveat that aid works better where existing institutions are stronger (more below) or that aid has diminishing returns. Others believe that there is no relationship or that there is insufficient evidence. Two particularly accessible summaries: A Primer on Foreign Aid (from the optimistic side) and Macro Aid Effectiveness Research: a Guide for the Perplexed (from the skeptical side).

There are no established broad patterns in the sorts of environments where aid has worked, though many believe that aid works better where existing institutions are stronger.

A Primer on Foreign Aid states,

the view that aid works better (or in a stronger version, aid works only) in countries with good policies and institutions has become the conventional wisdom among donors, partly based on [empirical] research and partly due to development practitioners that believe this to be the case based on their own experience.

However, as this paper acknowledges, the empirical research has been questioned repeatedly (as in this paper, which claims that the statistical work underlying most such claims is excessively fragile). In addition, Millions Saved (the success stories compilation referred to above) gives examples that “Success is possible even in the world’s most underdeveloped and remote regions, in the face of grinding poverty and weak health systems” (quote available at this page).

One caveat to keep in mind about all of this analysis is that most academic literature focuses on official aid flows - aid from developed-world governments (or multilateral institutions such as the World Bank), which usually goes through developing-world governments. This sort of aid is different in many ways from private donations going through nonprofits.