The GiveWell Blog

Measurement is not as common as it should be. Why?

(updated on: ) | by Holden

The idea that there should be more measurement appears to be one of the points of widest agreement in the literature on aid. But we believe that agreement in principle is unlikely to mean much until donors (both large and small) act on it. It isn’t enough to request better information; we need to reserve our funds for those who produce it.

This post has two sections. First we give a sample of quotes from a broad set of people and institutions, showing how widespread the call for better measurement is. We then discuss why agreement isn’t enough.

Widespread calls for more and better measurement

From what we’ve seen, the wish for more and better measurement is a near-universal theme in discussions of how to improve international aid. Below is a sample of relevant quotes.

Abhijit Banerjee, Director of the Poverty Action Lab (which used to employ one of our Board members), puts better evaluation at the heart of his 2007 book, Making Aid Work:

The reason [past success stories such as the eradication of smallpox] succeeded, I suspect, is that they started with a project that was narrowly defined and well founded. They were convinced it worked, they could convince others, and they could demonstrate and measure success. Contrast this with the current practice in development aid; as we have seen, what goes for best practice is often not particularly well founded. (pages 22-23)

William Easterly argues for the importance and centrality of evaluation in The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good:

[S]ome equally plausible interventions work and others don’t. Aid agencies must be constantly experimenting and searching for interventions that work, verifying what works with scientific evaluation. For learning to take place, there must be information. The aid agencies must carefully track the impact of their projects on poor people using the best scientific tools available, and using outside evaluators to avoid the self-interest of project managers. (page 374)
…
Think of the great potential for good if aid agencies probed and experimented their way toward effective interventions … Think of the positive feedback loop that could get started as success was rewarded with more resources and expanded further. (page 383)

Jeffrey Sachs, former director of the United Nations Millennium Project (and known for disagreeing with Easterly on many issues – a partial set of debates is available here), calls for more evaluation in The End of Poverty: Economic Possibilities for Our Time:

Much clearer targets of what is to be achieved must accompany a major increase of spending. Every [Millennium Development Goal]-based poverty reduction strategy should be supported by quantitative benchmarks tailored to national conditions, needs, and data availability … Right from the start, the … poverty reduction strategy should prepare to have the investments monitored and evaluated. Budgets and mechanisms for monitoring and evaluation should be essential parts of the strategies. (pages 278-9)

The Center for Global Development created a working group (with support from major foundations) specifically to examine why “very few programs benefit from studies that could determine whether or not they actually made a difference.” Its report provides further argument for why more evaluation is necessary:

Rigorous studies of conditional cash transfer programs, job training, and nutrition interventions in a few countries have guided policymakers to adopt more effective approaches, encouraged the introduction of such programs to other places, and protected large-scale programs from unjustified cuts. By contrast, a dearth of rigorous studies on teacher training, student retention, health financing approaches, methods for effectively conveying public health messages, microfinance programs, and many other important programs leave decisionmakers with good intentions and ideas, but little real evidence of how to effectively spend resources to reach worthy goals. (page 2)

The concern is not limited to researchers and think tanks: it’s one of the primary elements of the Paris Declaration on Aid Effectiveness, which emerged from a major meeting of “development officials and ministers from ninety one countries, twenty six donor organizations and partner countries, representatives of civil society organizations and the private sector.” The declaration states that aid recipients should “Endeavour to establish results-oriented reporting and assessment frameworks that monitor progress against key dimensions of the national and sector development strategies” (pg 8 ), and that donor governments should “Link country programming and resources to results” (pg 8 ). One of its 12 key indicators of progress is an increase in the “Number of countries with transparent and monitorable performance assessment frameworks” (page 10).

For our part, we strongly believe in the importance of measurement, particularly for public charities soliciting individual donations. Some of our reasoning echoes the arguments given above – that helping people is hard, that intuition is a poor guide to what works, and that measurement is necessary for improvement. We also feel the argument is even stronger for the particular area we’re focused on: helping people who have a few thousand dollars and a few hours, but ultimately know very little about the organizations they’re funding and the people they’re trying to help. Formal measurement is necessary for individual donors to hold charities accountable.

Why agreement doesn’t translate to action

The quotes above share a commitment not just to the general usefulness of measurement, but to the idea that there should be more of it than there is. This raises the question taken up by Lant Pritchett in “It pays to be ignorant: a simple political economy of rigorous program evaluation” (PDF): why is quality evaluation so rare, or as he puts it, “How can [the] combination of brilliant well-meaning people and ignorant organization be a stable equilibrium?”

Pritchett argues (pages 33-34) that the scarcity of evaluation can’t be explained by concerns such as expense, practical difficulty, or ethical concerns. He conjectures, instead, that the lack of evaluation is due to strategic behavior by those closest to the programs. These “advocates” tend to be strongly committed to the programs they work on, to the point where their behavior is guided more by trying to get more funding for these programs than trying to get new information on whether they work. According to Pritchett’s model, when donors do not demand rigorous evaluation, “advocates may choose ignorance over public knowledge of true program efficacy … even if it means they too must operate somewhat in the dark” (page 7).

In our view, the most compelling support for Pritchett’s model is his claim that

a huge number of evaluations are started and very few are finished, written up, and publicized. This evaluation attrition is too large to be consistently “bad planning” and is more likely strategic behavior.

This claim matches extremely well with our own observations of charities’ materials. The grant applications we’ve received (available online) frequently give reasonable-sounding plans for future evaluation, but rarely have results from past evaluations available.

We believe that foundations tend to focus on innovation as opposed to past results, while individual donors currently don’t have the information/ability/interest to hold agencies accountable in any way. We know less about the incentives and practices of large aid agencies (governments, the World Bank, etc.) but find it possible that they are driven more by politics and appearances by humanitarian results.

In other words, funders are not forcing evaluation to happen, and until they do, there’s little reason to expect improved/increased measurement. Agreement isn’t enough – if we want better information on what works, we need to commit with our dollars, not just our beliefs. As an individual donor, you have a key role – perhaps the key role – to play.

Review of The Life You Can Save, by Peter Singer

(updated on: ) | by Holden

The Life You Can Save went on sale in the U.S. on Monday. First, disclosures: the book prominently features GiveWell, a portion of the book’s proceeds are being donated to GiveWell, and I was sent an advance copy. I have strong incentives to encourage people to read and buy the book.

So let me start with a reason not to read it: it will make you uncomfortable. It certainly made me uncomfortable. It started by asking me a simple question – would I sacrifice time and money to save a stranger’s life? If so, why don’t I give more of my income to charity? – and pounded away relentlessly, tearing apart every excuse I had until I was left with “I’m really selfish.”

I’ve appreciated many books for making me feel scared, or angry, or sad. Now there’s one to make me feel personally guilty. (How’s that for a blurb?)

Of course the goal of the book isn’t to make people feel guilty, it’s to get them to give a lot (even if not as much as they, strictly speaking, could). And unlike the IRS, Prof. Singer doesn’t see supporting the local museum as equivalent to saving children’s lives. He’s specifically advocating more giving to developing-world aid, a goal we strongly agree with (as our research agenda demonstrates). You could think of this book as an End of Poverty on a personal rather than global scale – instead of arguing that the international community has the power to end poverty, it argues that you have the power (and thus the responsibility) to save a life.

But can a donation really save a life?

As with The End of Poverty, the moral argument depends on factual questions, and meets some skepticism from William Easterly, who argues – partly from GiveWell’s experience trying to find great charities – that saving a life is not as simple as it’s often made to sound.

There is merit to this. We’ve put a lot of effort by now into finding charities you can be confident in, and we still consider it an open question whether a $1000 donation really translates to a saved life. We estimate that it can in PSI’s case, but there are all kinds of room for uncertainty.

For example. To me the biggest questions with PSI are, (1) Is it getting its subsidized life-saving materials (mostly condoms and insecticide-treated nets) to people who need them, rather than to people who don’t? (2) Are these people consistently and correctly using the materials? One of the reasons I really like PSI is it seems very concerned with these two questions, and attempts to collect data specifically on them; the data it makes available imply success. On the other hand, a lot of monitoring and evaluation isn’t getting done (see the research scorecard, which to its credit PSI makes public), and none of it appears externally audited. How reliable is this data? How representative is the information we have?

And that’s PSI, our current top-recommended charity. Even if $1000 can save a life, your $1000 isn’t unless it gets used well. There’s no charity that makes me even 90% confident this is happening, and with the “average” charity I’d bet that it isn’t.

We can do more – not just give more

However, the bottom line is that I don’t think these concerns mean that Prof. Singer’s challenge can be dismissed. For one thing, even if 90% of PSI’s activities accomplish nothing and the other 10% are in line with our impressions, that’s still $10,000 per life saved – enough for the moral argument to remain very relevant, in my opinion. Based on the limited information we have, it appears that donating to our recommended charities likely is saving lives at some relatively good rate. It might be more uncertain and probabilistic than pulling a drowning child out of the water, but it’s still a compelling value for your money.

And the other issue is that there are more charities out there to be examined, and more improvement to be had from holding them accountable. As Prof. Easterly acknowledges, there are many proven life-saving programs. There may not be infinite room to expand these programs; these programs may not be able to end poverty by themselves; but they can absorb at least a few million more dollars. And that does mean that nearly all of us could be doing more to save (or change) lives than we are.

It’s just that “doing more” has to mean more than “giving more.” Picking your charity – and doing your part in holding it accountable – is at least as important as giving generously. We’re trying to make this task easier for time-strapped donors: if you put credence in our analysis, it can mean simply basing your giving on our recommendations (informally, or formally via GiveWell Advance Donation).

Bottom line

Unlike many “give more” advocates that target only dollars spent, Prof. Singer recognizes the challenge of translating generosity into results (hence his interest in GiveWell, as well as J-PAL, which we’re big fans of). His book challenges you to give more and give better. Neither of these is easy … nor is reading The Life You Can Save. But they’re worth it, because even for an individual donor, saving a life is within reach.

The Center for High Impact Philanthropy

(updated on: ) | by Elie

We’re very excited about the Center for High Impact Philanthropy, which recently released two reports: one focusing on increasing equality of opportunity in the United States and one on combating Malaria in the developing world. (H/t Alanna Shaikh)

We’ve read the full-length reports (available via email request to the Center) and we’re excited because:

  • Their reports focus on cost-effectiveness in human terms. Many global health reports focus heavily on the the DALY’s metric, which can make it hard for donors to see the human impact of their donations.
  • Their reports deal directly with issues of evidence of effectiveness, aiming to cite rigorous evidence where it exists.
  • They recommend specific charities and programs so that donors can act on the Center’s findings.

We’ve only briefly looked at these two reports, but based on what we’ve seen, we’re looking forward to their future publications and recommend that interested donors check out their work.

Preview report

| by Holden

Now linked from the front page of GiveWell.net is a preview of our 2008-2009 report. The main content of the report so far is a review of the Carter Center (discussed in previous blog posts here and here) as well as information on the track records of the programs it runs and the diseases it targets.

There is much more on the way, including the continuation of my series of overviews of general issues in developing-world aid (earlier entries here and here). For now, though, the review of the Carter Center (and accompanying materials) will give a strong sense of our basic structure, approach, and criteria, which have changed significantly since our 2007-2008 report. We are eager for feedback.

Preview of 2008-2009 Report

Learning from a small failure

(updated on: ) | by Holden

Global Health Report:

A computer science group from the State University of New York at Stonybrook presented three applications or “apps,” that is to say mini-computer programs, that they had designed for use on no-frills mobile phones owned by women working in the informal Senegalese economy. The pilot tests for two of the apps—a dictionary and a book-keeping calculator—were deemed successes. The third app—for measuring profit and loss—was judged a failure.

…

The pilot was a failure because the fish sellers found the mobile phone profit-and-loss calculator useless. The Stonybrook group did not learn why the app was useless, however, until a second round of testing in which one of the Senegalese computer science students happened to have a grandmother who was a fish seller. After talking with the fish sellers, he explained to the Stonybrook group that all the prices for both fresh fish and dried fish are fixed. Since everyone charges the same price for fish (one for dried, the other for fresh) on any given day, there is no way for the women to wait until the price is right.

It’s a small example, but we wish we saw more stories like this – public sharing of people trying a program, critically assessing it, and learning from what doesn’t work.

The root causes of poverty

(updated on: ) | by Holden

GiveWell generally focuses on the question of how to get “bang for your buck” as a donor – help as many people as possible, as much as possible. Against this approach, one might seek to factor in the potential of a program to get at the “root causes” of poverty, and start – or be part of – a chain reaction that ends poverty at the country or even world level. (One example of such reasoning in nonprofit marketing is here.)

Below is our take on the following broad question:

Why have some parts of the world emerged from poverty while others haven’t? How can financial aid from developed nations best be directed to cause large-scale emergence from poverty?

To us the key points are:

Past emergences from poverty have taken many different forms; there are no clear/consensus patterns or formulas in these stories.

This statement – which we have seen little to no literature contradicting – is well illustrated by the work of two major commissions, each of which set out to find patterns in the history of economic development.

A 1993 World Bank study on the rapid growth of 8 countries in East Asia concluded (overview; similar content on page 366):

The study attempts to explain East Asia’s success and to develop a model of rapid growth with equity. It finds that the diversity of experience, the variety of institutions, and the variations in policies among the [high-performing Asian economies] does not allow a model to be developed.

A more recent (2008) study, “led by 19 experienced policymakers and two Nobel prize-winning economists” (from the press release), contains similar caveats (from page 2):

The report identifies some of the distinctive characteristics of high-growth economies and asks how other developing countries can emulate them. It does not provide a formula for policy makers to apply—no generic formula exists. Each country has specific characteristics and historical experiences that must be reflected in its growth strategy. But the report does offer a framework that should help policy makers create a growth strategy of their own. It will not give them a full set of answers, but it should at least help them ask the right questions. Fast, sustained growth does not happen spontaneously. It requires a long-term commitment by a country’s political leaders, a commitment pursued with patience, perseverance, and pragmatism.

There are many different prescriptions for the actions most likely to end poverty.

Various scholars propose large-scale plans – based on theories of the root causes of poverty – for ending (or drastically reducing) poverty. However, it should be noted that their plans (a) are very different from each other; (b) tend to be highly multidimensional and to depend heavily on governments and/or international institutions. (In other words, few put a single “silver bullet” intervention at the heart of their plans.) Three prominent examples:

  • Jeffrey Sachs (The End of Poverty) argues that poverty itself is self-reinforcing: when people and governments have low enough income, they cannot make the necessary investments to create strong future growth (see pages 245-250). His recommendations, presented in the UN publication Investing in Development, center around aid to developing-world governments, in support of multidimensional and country-specific poverty reduction strategies (page xx). Such aid is to be roughly doubled from 2003 levels by 2015 (page xxii); other recommendations include loosening trade restrictions and increasing relevant scientific research (page xxii).
  • Paul Collier (The Bottom Billion) believes the world’s poorest countries are caught in one or more of four “traps” (page 17): patterns of civil war (chapter 2), “resource curses” in which large natural resource wealth prevents healthy economic development (page 39), the condition of being “landlocked with bad neighbors” (53), and bad governance in small countries (65). His agenda (177-183), like Sachs’s, includes an emphasis on free trade, but it also includes major roles for military intervention to deal with conflict as well as international laws and charters to hold governments accountable. Although he supports aid to some governments (179), he specifically cautions against the dangers of too much aid in some cases (page 181), and proposes circumventing governments through “independent service authorities” in others (177, 179).
  • Peter Timmer (Agriculture and Pro-Poor Growth: an Asian Perspective) argues that “No country has been able to sustain a rapid transition out of poverty without raising productivity in its agricultural sector” (page 3) and recommends a set of interventions focusing on rural areas, particularly agriculture (pages 29-30).

There are reasons to see health aid as a promising approach to reducing poverty. There are also reasons to see it as neither necessary nor sufficient for this goal.

The idea that improving health would lead to improved productivity, and thus less poverty, is one that makes intuitive sense and has some suggestive evidence to support it. Working Group 1 of the WHO Commission on Macroeconomics and Health (PDF) summarizes many different kinds of evidence including cross-country analysis, studies on the productivity of people receiving nutrition supplements, and analysis of changes in calories available over time (see pages 5-12 for an overview, although we are looking for a clearer and more complete review on this topic).

But there is little reliable guide to how much economic improvement can be expected to come directly from health improvement, and even strong advocates of health aid such as the Commission on Macroeconomics and Health do not see health aid as sufficient (by itself) to end poverty. (See pages 28-29 of the commission’s final report.) There is also no consensus that health aid is necessary; some see past emergences as having been led by agricultural improvements (such as Timmer, discussed above) and/or having been accomplished largely without external assistance of any kind (as William Easterly does – see page 347 of White Man’s Burden).

Bottom line

No single theory of the “root causes of poverty” is supported by overwhelming evidence or broad expert consensus. We don’t find any compelling enough, or relevant enough to what individual donors can do, to compete strongly with the goal of improving individuals’ lives – a goal that could itself be the best approach to speeding the end of poverty, particularly if you believe (as scholars such as William Easterly do) that the emergence of nations is most likely to be homegrown.

There are many proven, cost-effective, scalable ways to significantly improve people’s lives. We feel that adding to one is the best use of an individual donor’s funds.