As part of our current work on developing-world aid, we’ve completed a preliminary report on The Stop TB Partnership. We still have more work to do, but want to share what we’ve learned thus far. Here’s what’s available:
The Knowledge is Power Program is one of our current recommended charities, but I think that Sarah Mosle’s critique in Slate is very much worth keeping in mind.
Mosle writes:
While KIPP does have outreach efforts to broaden its applicant pool, only the most determined parents are likely to respond to … sign KIPP’s demanding contract. This dedication suggests a higher value on education within these families, and thus kids better able or willing to learn. And the weakest students, not surprisingly, get disproportionately winnowed. In KIPP’s schools in the San Francisco Bay Area, for example, the worst-performing kids have dropped out (or been expelled) in greater numbers in the higher grades; the result has been to inflate the schools’ grade-to-grade improvement.
We agree that the superior performance of KIPP’s students can’t be taken fully at face value, because they may not be a truly representative set of disadvantage students. Our analysis concludes that KIPP most likely is making a difference for the students that it serves, despite these concerns (and Mosle thinks so as well).
However, just because KIPP is making a difference for the students it serves doesn’t mean its model can be fully generalized to close the achievement gap. For one thing, it isn’t clear how many teachers can be found that are at the caliber KIPP aims for. For another, KIPP appears to be aimed at a particular kind of student. I think Mosle’s closing concern is right on target:
But since the biggest debate about KIPP, on both the ideological left and right, is whether or not its methods can work for all disadvantaged children (instead of just a handful of self-selecting families), why wouldn’t it—and its financial, ideological, and media backers—have a strong interest in answering this question once and for all by taking on an entire urban area or even, for that matter, a single neighborhood as, say, Geoffrey Canada has tried to do in Harlem with his Harlem’s Children’s Zone?
There’s something perversely evasive about KIPP’s opening up just one school in Dallas, one school in Albany, N.Y., one school in Oakland, Calif., one school in Charlotte, N.C., one school in Nashville, Tenn., and so on—as if the program recognizes that its best chance at success is to be the exception rather than the rule in any city where it operates.
I believe anyone pointing to KIPP as “the path to closing the achievement gap” is being far too optimistic, although KIPP is a promising way to improve outcomes for the individuals it serves.
Additional GiveWell materials related to KIPP:
There are many studies attempting to gauge microloans’ impact on borrowers, but most suffer from the problem of selection bias: by comparing participants and non-participants, they may be picking up other differences between these groups (for example, people who participate in microloan programs may be wealthier to begin with, so a study showing that they have higher incomes does not really show that microloans help).
One of the better-known attempts to avoid this problem was a 1997 study that attempted to take advantage of a “landholding requirement”: a program that allowed people to borrow only if they held less than a half-acre of land. This study is one of the few empirically encouraging studies of microloan programs, as we discuss here, although there was some dispute over whether the landholding requirement was strictly enforced.
New analysis by the Center for Global Development’s David Roodman shows that even though the microfinance program attempted to target people with less land, in fact people with more land borrowed significantly more. I agree with Roodman’s general conclusion:
This is not necessarily bad: serving richer clients with larger loans may make it more economical to reach poorer people in the same villages–an example of cross-subsidization. And someone whose main asset is an acre or two or riceland in rural Bangladesh is hardly rich by western standards–and might even make better use of the credit. Still, the “mistargeting” of landed families contradicts the public image of microfinance in Bangladesh as targeting the poorest, and probably needs to be better understood.
Additional GiveWell materials related to microfinance:
A couple of recent web/media appearances:
In his Annual Letter, Bill Gates describes the “Green Revolution”:
Almost every country that has become wealthy started with a huge increase in farming productivity. Chart 4 shows the increase in output per acre for various grains, including wheat, corn, and rice, in the United States, India, China, and Africa since 1961. This dramatic increase in output—more than three times—is often called the Green Revolution.
The Green Revolution, and the Rockefeller Foundation’s role in the research that enabled it, is frequently cited as one of philanthropy’s great success stories(1) and a huge contributor to enormous reductions in poverty.(2) Yet as Gates continues, “Africa jumps out as the only case where this [revolution] has not taken place.” Why?
To Gates, the answer appears to come down to insufficient investment:
African countries have widely varying climate conditions, and there hasn’t been the same investment in creating the seeds that fit those conditions. Because agriculture is an essential part of economic growth for most African countries, we are working with others to fund a “Green Revolution for Africa” and other areas that could benefit from this kind of investment.
Gates gives the impression that bringing the Green Revolution in Africa is mostly a matter of repeating what’s worked elsewhere, which would make it an excellent fit for our priorities. But from the rough analysis we’ve done, it appears that there has been at least as much effort to bring about a Green Revolution in Africa as elsewhere, and the obstacles in this area are specific and significant.
CGIAR funding
The Consultative Group on International Agricultural Research appears to have been the main vehicle for philanthropic funding of relevant research.(3) Using data from its website, we put together the chart below showing how much has been spent at its centers in (sub-Saharan) Africa as opposed to its other centers. The proportion was consistently been between 25% and 30% from 1972-2003:(4)
For context, sub-Saharan Africa accounted for 20-30% of the world’s extremely poor in 2003, up from 14-19% in 1990.(5) It certainly seems from this rough cut that funding for relevant research in Africa was in line with funding for relevant research in the rest of the world.
Norman Borlaug
Norman Borlaug is often credited with (and won the 1970 Nobel Peace Prize for) a leading role in the research that made the Green Revolution possible.(6) The transcript of a 2006 Center for Global Development event implies the following about his relative efforts in different areas:
Comparable efforts; disappointing results
There are many factors that may make a Green Revolution in Africa difficult or impossible to bring about, including:(7)
None of this means that bringing the Green Revolution to Africa is necessarily impossible, or that aiming funding at this goal is necessarily futile. But it’s important to recognize that this goal is a formidable challenge for which no strong precedent exists.
The prospect of an African Green Revolution is extremely appealing, but we don’t feel that this sort of investment can ultimately be counted as “proven and scalable,” and we don’t feel it’s as well-suited for individual donors as many health interventions (which are both proven and likely repeatable, and have simply not been funded enough to reach full coverage – more on this in a future post).
(1) See, for example:
(2) “The green revolution, which accelerated growth from the 1960s, beginning in India and Indonesia, was a major factor reducing poverty in Asia, as documented by numerous studies (see, for example, Rosegrant and Hazell 2000; Timmer 2002; Lipton 2004; Datt and Ravallion 1998a, 1998b). ” From “Agriculture, Rural Development, and Pro-poor growth” (World Bank 2005) pg 15.
(3) Based on a reading of the Rockefeller Foundation’s history (note 1 gives two examples of the Rockefeller Foundation’s being credited with a primary role in the Green Revolution).
(4) Data, sources, and calculations available here (XLS).
(5) Based on the proportion of people living on $1/day or less and $2/day or less, as reported on page 60 of the World Bank’s 2007 Global Economic Prospects report.
(6) See Borlaug’s Nobel Peace Prize bio and the opening statements/summary of the Center for Global Development’s event on “The Prospects of Bringing a Green Revolution to Africa”.
(7)Sources:
The idea that there should be more measurement appears to be one of the points of widest agreement in the literature on aid. But we believe that agreement in principle is unlikely to mean much until donors (both large and small) act on it. It isn’t enough to request better information; we need to reserve our funds for those who produce it.
This post has two sections. First we give a sample of quotes from a broad set of people and institutions, showing how widespread the call for better measurement is. We then discuss why agreement isn’t enough.
Widespread calls for more and better measurement
From what we’ve seen, the wish for more and better measurement is a near-universal theme in discussions of how to improve international aid. Below is a sample of relevant quotes.
Abhijit Banerjee, Director of the Poverty Action Lab (which used to employ one of our Board members), puts better evaluation at the heart of his 2007 book, Making Aid Work:
The reason [past success stories such as the eradication of smallpox] succeeded, I suspect, is that they started with a project that was narrowly defined and well founded. They were convinced it worked, they could convince others, and they could demonstrate and measure success. Contrast this with the current practice in development aid; as we have seen, what goes for best practice is often not particularly well founded. (pages 22-23)
William Easterly argues for the importance and centrality of evaluation in The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good:
[S]ome equally plausible interventions work and others don’t. Aid agencies must be constantly experimenting and searching for interventions that work, verifying what works with scientific evaluation. For learning to take place, there must be information. The aid agencies must carefully track the impact of their projects on poor people using the best scientific tools available, and using outside evaluators to avoid the self-interest of project managers. (page 374)
…
Think of the great potential for good if aid agencies probed and experimented their way toward effective interventions … Think of the positive feedback loop that could get started as success was rewarded with more resources and expanded further. (page 383)
Jeffrey Sachs, former director of the United Nations Millennium Project (and known for disagreeing with Easterly on many issues – a partial set of debates is available here), calls for more evaluation in The End of Poverty: Economic Possibilities for Our Time:
Much clearer targets of what is to be achieved must accompany a major increase of spending. Every [Millennium Development Goal]-based poverty reduction strategy should be supported by quantitative benchmarks tailored to national conditions, needs, and data availability … Right from the start, the … poverty reduction strategy should prepare to have the investments monitored and evaluated. Budgets and mechanisms for monitoring and evaluation should be essential parts of the strategies. (pages 278-9)
The Center for Global Development created a working group (with support from major foundations) specifically to examine why “very few programs benefit from studies that could determine whether or not they actually made a difference.” Its report provides further argument for why more evaluation is necessary:
Rigorous studies of conditional cash transfer programs, job training, and nutrition interventions in a few countries have guided policymakers to adopt more effective approaches, encouraged the introduction of such programs to other places, and protected large-scale programs from unjustified cuts. By contrast, a dearth of rigorous studies on teacher training, student retention, health financing approaches, methods for effectively conveying public health messages, microfinance programs, and many other important programs leave decisionmakers with good intentions and ideas, but little real evidence of how to effectively spend resources to reach worthy goals. (page 2)
The concern is not limited to researchers and think tanks: it’s one of the primary elements of the Paris Declaration on Aid Effectiveness, which emerged from a major meeting of “development officials and ministers from ninety one countries, twenty six donor organizations and partner countries, representatives of civil society organizations and the private sector.” The declaration states that aid recipients should “Endeavour to establish results-oriented reporting and assessment frameworks that monitor progress against key dimensions of the national and sector development strategies” (pg 8 ), and that donor governments should “Link country programming and resources to results” (pg 8 ). One of its 12 key indicators of progress is an increase in the “Number of countries with transparent and monitorable performance assessment frameworks” (page 10).
For our part, we strongly believe in the importance of measurement, particularly for public charities soliciting individual donations. Some of our reasoning echoes the arguments given above – that helping people is hard, that intuition is a poor guide to what works, and that measurement is necessary for improvement. We also feel the argument is even stronger for the particular area we’re focused on: helping people who have a few thousand dollars and a few hours, but ultimately know very little about the organizations they’re funding and the people they’re trying to help. Formal measurement is necessary for individual donors to hold charities accountable.
Why agreement doesn’t translate to action
The quotes above share a commitment not just to the general usefulness of measurement, but to the idea that there should be more of it than there is. This raises the question taken up by Lant Pritchett in “It pays to be ignorant: a simple political economy of rigorous program evaluation” (PDF): why is quality evaluation so rare, or as he puts it, “How can [the] combination of brilliant well-meaning people and ignorant organization be a stable equilibrium?”
Pritchett argues (pages 33-34) that the scarcity of evaluation can’t be explained by concerns such as expense, practical difficulty, or ethical concerns. He conjectures, instead, that the lack of evaluation is due to strategic behavior by those closest to the programs. These “advocates” tend to be strongly committed to the programs they work on, to the point where their behavior is guided more by trying to get more funding for these programs than trying to get new information on whether they work. According to Pritchett’s model, when donors do not demand rigorous evaluation, “advocates may choose ignorance over public knowledge of true program efficacy … even if it means they too must operate somewhat in the dark” (page 7).
In our view, the most compelling support for Pritchett’s model is his claim that
a huge number of evaluations are started and very few are finished, written up, and publicized. This evaluation attrition is too large to be consistently “bad planning” and is more likely strategic behavior.
This claim matches extremely well with our own observations of charities’ materials. The grant applications we’ve received (available online) frequently give reasonable-sounding plans for future evaluation, but rarely have results from past evaluations available.
We believe that foundations tend to focus on innovation as opposed to past results, while individual donors currently don’t have the information/ability/interest to hold agencies accountable in any way. We know less about the incentives and practices of large aid agencies (governments, the World Bank, etc.) but find it possible that they are driven more by politics and appearances by humanitarian results.
In other words, funders are not forcing evaluation to happen, and until they do, there’s little reason to expect improved/increased measurement. Agreement isn’t enough – if we want better information on what works, we need to commit with our dollars, not just our beliefs. As an individual donor, you have a key role – perhaps the key role – to play.