Global Health Report:

A computer science group from the State University of New York at Stonybrook presented three applications or “apps,” that is to say mini-computer programs, that they had designed for use on no-frills mobile phones owned by women working in the informal Senegalese economy. The pilot tests for two of the apps—a dictionary and a book-keeping calculator—were deemed successes. The third app—for measuring profit and loss—was judged a failure.

…

The pilot was a failure because the fish sellers found the mobile phone profit-and-loss calculator useless. The Stonybrook group did not learn why the app was useless, however, until a second round of testing in which one of the Senegalese computer science students happened to have a grandmother who was a fish seller. After talking with the fish sellers, he explained to the Stonybrook group that all the prices for both fresh fish and dried fish are fixed. Since everyone charges the same price for fish (one for dried, the other for fresh) on any given day, there is no way for the women to wait until the price is right.

It’s a small example, but we wish we saw more stories like this - public sharing of people trying a program, critically assessing it, and learning from what doesn’t work.

GiveWell generally focuses on the question of how to get “bang for your buck” as a donor - help as many people as possible, as much as possible. Against this approach, one might seek to factor in the potential of a program to get at the “root causes” of poverty, and start - or be part of - a chain reaction that ends poverty at the country or even world level. (One example of such reasoning in nonprofit marketing is here.)

Below is our take on the following broad question:

Why have some parts of the world emerged from poverty while others haven’t? How can financial aid from developed nations best be directed to cause large-scale emergence from poverty?

To us the key points are:

Past emergences from poverty have taken many different forms; there are no clear/consensus patterns or formulas in these stories.

This statement - which we have seen little to no literature contradicting - is well illustrated by the work of two major commissions, each of which set out to find patterns in the history of economic development.

A 1993 World Bank study on the rapid growth of 8 countries in East Asia concluded (overview; similar content on page 366):

The study attempts to explain East Asia’s success and to develop a model of rapid growth with equity. It finds that the diversity of experience, the variety of institutions, and the variations in policies among the [high-performing Asian economies] does not allow a model to be developed.

A more recent (2008) study, “led by 19 experienced policymakers and two Nobel prize-winning economists” (from the press release), contains similar caveats (from page 2):

The report identifies some of the distinctive characteristics of high-growth economies and asks how other developing countries can emulate them. It does not provide a formula for policy makers to apply—no generic formula exists. Each country has specific characteristics and historical experiences that must be reflected in its growth strategy. But the report does offer a framework that should help policy makers create a growth strategy of their own. It will not give them a full set of answers, but it should at least help them ask the right questions. Fast, sustained growth does not happen spontaneously. It requires a long-term commitment by a country’s political leaders, a commitment pursued with patience, perseverance, and pragmatism.

There are many different prescriptions for the actions most likely to end poverty.

Various scholars propose large-scale plans - based on theories of the root causes of poverty - for ending (or drastically reducing) poverty. However, it should be noted that their plans (a) are very different from each other; (b) tend to be highly multidimensional and to depend heavily on governments and/or international institutions. (In other words, few put a single “silver bullet” intervention at the heart of their plans.) Three prominent examples:

• Jeffrey Sachs (The End of Poverty) argues that poverty itself is self-reinforcing: when people and governments have low enough income, they cannot make the necessary investments to create strong future growth (see pages 245-250). His recommendations, presented in the UN publication Investing in Development, center around aid to developnig-world governments, in support of multidimensional and country-specific poverty reduction strategies (page xx). Such aid is to be roughly doubled from 2003 levels by 2015 (page xxii); other recommendations include loosening trade restrictions and increasing relevant scientific research (page xxii).
• Paul Collier The Bottom Billion) believes the world’s poorest countries are caught in one or more of four “traps” (page 17): patterns of civil war (chapter 2), “resource curses” in which large natural resource wealth prevents healthy economic development (page 39), the condition of being “landlocked with bad neighbors” (53), and bad governance in small countries (65). His agenda (177-183), like Sachs’s, includes an emphasis on free trade, but it also includes major roles for military intervention to deal with conflict as well as international laws and charters to hold governments accountable. Although he supports aid to some governments (179), he specifically cautions against the dangers of too much aid in some cases (page 181), and proposes circumventing governments through “independent service authorities” in others (177, 179).
• Peter Timmer (Agriculture and Pro-Poor Growth: an Asian Perspective) argues that “No country has been able to sustain a rapid transition out of poverty without raising productivity in its agricultural sector” (page 3) and recommends a set of interventions focusing on rural areas, particularly agriculture (pages 29-30).

There are reasons to see health aid as a promising approach to reducing poverty. There are also reasons to see it as neither necessary nor sufficient for this goal.

The idea that improving health would lead to improved productivity, and thus less poverty, is one that makes intuitive sense and has some suggestive evidence to support it. Working Group 1 of the WHO Commission on Macroeconomics and Health (PDF) summarizes many different kinds of evidence including cross-country analysis, studies on the productivity of people receiving nutrition supplements, and analysis of changes in calories available over time (see pages 5-12 for an overview, although we are looking for a clearer and more complete review on this topic).

But there is little reliable guide to how much economic improvement can be expected to come directly from health improvement, and even strong advocates of health aid such as the Commission on Macroeconomics and Health do not see health aid as sufficient (by itself) to end poverty. (See pages 28-29 of the commission’s final report.) There is also no consensus that health aid is necessary; some see past emergences as having been led by agricultural improvements (such as Timmer, discussed above) and/or having been accomplished largely without external assistance of any kind (as William Easterly does - see page 347 of White Man’s Burden).

Bottom line

No single theory of the “root causes of poverty” is supported by overwhelming evidence or broad expert consensus. We don’t find any compelling enough, or relevant enough to what individual donors can do, to compete strongly with the goal of improving individuals’ lives - a goal that could itself be the best approach to speeding the end of poverty, particularly if you believe (as scholars such as William Easterly do) that the emergence of nations is most likely to be homegrown.

There are many proven, cost-effective, scalable ways to significantly improve people’s lives. We feel that adding to one is the best use of an individual donor’s funds.

While Elie’s been investigating the Carter Center, I’ve been scanning literature (mostly academic) on general questions about aid: what has worked in the past? What’s promising for the future? etc.

Rather than trying to come to an independent conclusion on each debate, I’ve been trying to establish which beliefs are supported by evidence that is largely undisputed among scholars (and where there is no agreement, get a sense of what each side believes and what its most frequently pointed to evidence is).

At this point I’ve looked in a lot of places (though I’m still far from done) and I’m going to start sharing where we stand (and what we’re still missing) on various questions. First I’ll discuss what I’ve read about the track record of aid to developing-world areas.

The vast majority of the large-scale “success stories” I’ve seen come from health initiatives (particularly “vertical” health initiatives, i.e., large-scale campaigns against particular diseases).

• Millions Saved: Proven Successes in Global Health (published by the Center for Global Development) seems to be the most frequently cited reference by authors seeking to point to “success stories.” It lists 20 large-scale case studies including the global eradication of smallpox and the Carter Center-led campaign against guinea worm.
• There is also a success stories compilation by the World Health Organization, though these case studies don’t provide data sources (and haven’t responded to our request for them) and in many cases seem to be discussing smaller-scale initiatives.
• Jeffrey Sachs’s The End of Poverty lists ten “significant examples of programs that have been scaled up massively to remarkable success” (pgs 260-265). They are again dominated by health interventions. My notes on them are available here (via our public email list).

The main non-health “success stories” I’m aware of:

• Philanthropic funding of agricultural research is often credited with a major role in catalyzing the emergence from poverty of many countries in Asia and elsewhere (an emergence often referred to as the “Green Revolution” because of the prominent role of productivity gains in agriculture). So far, efforts to expand the “Green Revolution” to Africa have largely been disappointing (overview on pgs 66-72 of Can the West Save Africa? (PDF) by William Easterly).
• There is “suggestive” evidence that education aid has resulted in increasing primary enrollments, though the effect of this increasing enrollment on growth/standard of living/etc. is unclear (overview on pgs 47-53 of Can the West Save Africa? (PDF) by William Easterly).

I don’t believe that any of the sources cited above are fully comprehensive (or necessarily fully reliable) as lists of large-scale success stories. We’re still looking for more.

Aid has also had major failures - but insufficient monitoring and evaluation means that few are thoroughly documented or discussed.

• Dissatisfaction with the accomplishments of aid to date is fairly widespread. The latest report on the Millennium Development Goals shows mostly inadequate/less-than-hoped-for progress (and no progress in many cases). William Easterly argues that
  

  systematic testing would not just count the alleged ’success stories’ of aid, but also the larger number that got the same amount of aid as the ’success stories’ and failed: Guinea-Bissau, Somalia, The Gambia, Mali, Rwanda, Nicaragua, Burundi, Guyana, Zambia, the Central African Republic, Senegal, Suriname, Chad, Niger, Togo, Haiti, and so on. Further testing shows that these outcomes were not an artifact of selection bias or reverse causality.

• Yet while project-level evaluations provide scattered analysis of projects gone wrong (one example of a failed World Bank project here), we have not found well-documented “failure stories” along the lines of the “success stories” above - examining major humanitarian (as opposed to political) initiatives that had little or negative impact.
• Part of the reason may be the general lack of evaluation and documentation in international aid. The sentiments of this Center for Global Development paper, arguing that “very few programs benefit from studies that could determine whether or not they actually made a difference. This absence of evidence is an urgent problem,” are common; the Paris Declaration on Aid Effectiveness, an agreement between major aid agencies, includes an expressed commitment to more monitoring and evaluation.

I have the sense that failure stories in aid are common, but I continue to look for more concrete examples of what has gone wrong and how.

The relationship between aid and growth at the macro/country level has been extensively studied, but is not well established one way or the other.

Some believe in a moderate positive relationship, often with the caveat that aid works better where existing institutions are stronger (more below) or that aid has diminishing returns. Others believe that there is no relationship or that there is insufficient evidence. Two particularly accessible summaries: A Primer on Foreign Aid (from the optimistic side) and Macro Aid Effectiveness Research: a Guide for the Perplexed (from the skeptical side).

There are no established broad patterns in the sorts of environments where aid has worked, though many believe that aid works better where existing institutions are stronger.

A Primer on Foreign Aid states,

the view that aid works better (or in a stronger version, aid works only) in countries with good policies and institutions has become the conventional wisdom among donors, partly based on [empirical] research and partly due to development practitioners that believe this to be the case based on their own experience.

However, as this paper acknowledges, the empirical research has been questioned repeatedly (as in this paper, which claims that the statistical work underlying most such claims is excessively fragile). In addition, Millions Saved (the success stories compilation referred to above) gives examples that “Success is possible even in the world’s most underdeveloped and remote regions, in the face of grinding poverty and weak health systems” (quote available at this page).

One caveat to keep in mind about all of this analysis is that most academic literature focuses on official aid flows - aid from developed-world governments (or multilateral institutions such as the World Bank), which usually goes through developing-world governments. This sort of aid is different in many ways from private donations going through nonprofits.

In Part I, I laid out my case for The Carter Center, and why we think they’re worth investigating deeply. I also said that we’re not yet ready to recommend them because we have some unanswered questions. Here they are:

Relevance of your donation

When I give, I want to know that the organization needs my support and is going to accomplish more good (than they otherwise would have) because of my support. Because of The Carter Center’s strong track record, I wonder whether they can already effectively raise all the money they want. That is, I wonder whether money isn’t the bottleneck to serving more people. That question is supported by looking at Carter’s current financial situation. Revenues have risen roughly in-line with expenses over the past few years (see chart 1 below), maintaining their (relatively large) assets:expense ratio of 3:1 (see chart 2, below). All data comes from Carter Center annual reports, available on their website here.

Less-proven and less-monitored programs.

Many of The Carter Center’s programs have extremely strong independent evidence supporting them (e.g., their guinea worm, river blindness, lymphatic filariasis, and schistosomiasis programs). These programs also have the type of strong, consistent monitoring I mentioned in the previous post. However, the evidence case for some of their other programs is weaker (in particular their trachoma, agriculture, and public health training initiative), and the monitoring provided for those programs is both a) less consistent and b) less compelling than the monitoring provided for the others.

Running some less-proven projects doesn’t necessarily mean we won’t recommend them, especially if those programs are relatively small, but:

Lack of financial transparency

The Carter Center is one of the (if not the) most transparent organizations I’ve ever come across when it comes to monitoring their activities. Carter not only lays out what they do in extreme detail — for the intrepid, the most recent river blindness program report (PDF) is representative of Carter Center monitoring — they discuss program trade-offs they make, lessons they’ve learned, and potential future obstacles. But, they don’t publish any cost data on their website, other than the very broad and general 990 data and some sporadic program-specific information. To properly evaluate The Carter Center, we need to know the portion of the budget that’s allocated to any less-proven programs. In addition, we want tell you what your donation accomplishes to The Carter Center. Does $1,000 prevent 1 case of blindness? 2? 10? We need better information on costs to make that call.

Those are the questions we’re working on answering now. Whatever we find, you’ll get our answers soon.

Aid Watch on questionable claims of success against malaria:

Real victories against malaria would be great, but false victories can mislead and distract critical malaria efforts. Alas, Mr. and Mrs. Gates are repeating numbers that have already been discredited. This story of irresponsible claims goes back to a big New York Times headline on February 1, 2008: “Nets and New Drug Make Inroads Against Malaria,” which quoted Dr. Arata Kochi, chief of malaria for the WHO, as reporting 50-60 percent reductions in deaths of children in Zambia, Ethiopia, and Rwanda, and so celebrated the victories of the anti-malaria campaign. Alas, Dr. Kochi had rushed to the press a dubious report. The report was never finalized by WHO, it promptly disappeared, and its specific claims were contradicted by WHO’s own September 2008 World Malaria Report, by which time Dr. Kochi was no longer WHO chief of malaria.

Video with the Gateses available here.

When is a measured program-impact not a real impact? When it doesn’t last.

A study published today in PloS NTDs evaluated the impact of four doses of azithromycin (one every 6 months), and monitored trachoma prevalence throughout the drug administration period and for 2 years after the last dose.

In the first 24 months (from the start of the program through 6 months after the last treatment) prevalence decreased from 63.5% to 2.6%. In the 18 months after the last treatment, prevalence increased to 25.2%.

There was wide village-to-village variation in the prevalence before, during, and after treatment. While all villages ended up with lower infection rates of trachoma than they started with, in some (in particular village numbers 10 and 12 in this table), the infection rate 18 months post-treatment was close to pre-treatment levels.

Another reason to think that more monitoring is probably worth the cost.

I’ve spent a good part of the past month reviewing The Carter Center in depth. We found out about The Carter Center though referrals from an advisor and donor, and through the Gates Award for Global Health.

We chose to investigate The Carter Center further for four reasons:

1. Strong track record of success. The Carter Center has been credited with leading the global effort to eradicate guinea worm. In 1986, when Carter got involved, there were 3.5m annual cases of guinea worm. In 2008, there were fewer than 5,000. Guinea worm has been eliminated from 12 of the 18 countries that were once endemic. This chart shows the decline in cases from 1989-2007 (PDF). (More information about guinea worm and its symptoms is available on The Carter Center website.)
2. Strong program selection. Other Carter Center programs have strong evidence of effectiveness supporting them. The interventions they implement to control or eliminate neglected tropical disease including river blindness, trachoma, lymphatic filariasis, and schistosomiasis are recommended by the experts at the Disease Control Priorities Project.
3. Monitoring and evaluation. For its health programs, The Carter Center often monitors not only the number of drugs distributed in each region but disease prevalence directly. This monitoring isn’t evident for every single program or region, but it is far more consistent than any other complex organization we’ve seen before.
4. Transparency. All the information we’ve used to research The Carter Center is available publicly on their website. For example, the detailed program reports I mentioned above are available here. For an example of the type of monitoring conducted for its guinea worm program, see the latest monthly report linked here.

I still have some large, unanswered questions about The Carter Center, and we aren’t yet ready to recommend them.

PDF here (via Innovators for Poverty Action, whose research is featured). After discussing the Karlan/Zinman study showing benefits for loans (which we summarize here), it continues:

Karlan is the first to warn against extrapolating too much from a single experiment. “This is the last thing in the world that I would use to develop policy,” he warns. “You’ve got to replicate.”

The trouble is that the replication just isn’t happening. For all the optimism about microfinance – and the [Karlan/Zinman] experiment only encourages that optimism – it is striking how much we do not know about when it works, and why.

This matters because non-commercial microfinance projects often depend on donor subsidies. And while microfinance has a good reputation among development professionals, that doesn’t mean guaranteed access to those subsidies. Not everyone is convinced that a donor grant is best used to subsidise a loan rather than, say, pay directly for a primary school. More credible evaluation would help preserve the programmes that deserve to be preserved.

Already, solidly held beliefs about microfinance have been shaken. The “group liability” system, in which a group of borrowers guarantee one another’s loans, is still supposed by many to be the secret behind Grameen Bank’s low default rates. But a randomised trial in the Philippines conducted by Karlan and aWorld Bank economist, Xavier Gine, found that group liability was discouraging new customers without improving repayment rates. Grameen itself quietly dropped group liability some time ago.

Another sacred cow of microfinance is that women make best use of the money – the Grameen Bank says that 97 per cent of its borrowers are women. But another randomised trial, conducted in Sri Lanka by a team of researchers including David McKenzie of the World Bank, found that male borrowers seemed to make a far higher return on their capital. As with the ZaFinCo study, it’s just one experiment in one country. Yet it raises a worrying question: for how long will donors fund microfinance projects with so little compelling evidence about exactly what kinds of project really work?

It also discusses why the Karlan/Zinman’s results might not be generalizable:

[The experiment’s] clients were ideal customers for a commercial lender: they were city-dwellers and therefore cheap to reach; they were poor enough to want loans but rich enough that the loans were profitably chunky. A peasant farmer in Ethiopia or Sudan ticks none of those boxes: living in the middle of nowhere, he is expensive to reach and he is so poor that he can only afford tiny loans. Trapped in a barren economic ecosystem, he has no job that a ZaFinCo-style loan can help preserve, and no business prospects either. A mere loan will not catapult him into the ranks of the entrepreneurial class. Then there are the destitute, the disabled, the elderly and the orphans. Such people cannot repay loans at a rate that would cover costs. Heavy subsidies or outright grants would be needed. “All people are entrepreneurs,” says Muhammad Yunus. If only he were right.

And indeed, a more recent randomized trial of microfinance has preliminarily found far less encouraging results (discussed in this interview on Philanthropy Action, co-maintained by GiveWell Board member Tim Ogden).

A reader was good enough to send in a Lancet article (free registration required for full text) about a well-designed study of a combination microfinance/education program in South Africa.

Study design, strengths and weaknesses

A program consisting of both loans and group meetings was rolled out to 8 villages in rural South Africa, but the villages were randomly split into 4 that received it right away and 4 that received it 3 years later. Meetings included a curriculum that “covered topics including gender roles, cultural beliefs, relation ships, communication, intimate-partner violence, and HIV, and aimed to
strengthen communication skills, critical thinking, and leadership” (pg 1975).

Researchers hypothesized that (a) women in the loan groups would have fewer experiences of intimate-partner violence (presumably due to being financially/culturally more empowered); (b) this in turn would be connected with less unprotected sex in their households; (c) this in turn would slow the spread of HIV in their villages. A very ambitious theory of how to slow the spread of HIV - but to the researchers’ credit, they specified their hypotheses formally before conducting the study, as well as registering it on ClinicalTrials.gov. Combined with the use of randomization, this study had just about all the ingredients for avoiding the plague of publication bias.

A problem with the study, which the researchers partially acknowledge (pg 1981), is that it was only conducted in 8 villages total (4 receiving the program and 4 not receiving it). Therefore, it’s hard to say with confidence that any observed differences were due to the program as opposed to other differences between one randomly chosen set of 4 villages and another. Villages were similar on most observable characteristics, but very different on a few (see pg 1980).

Results

The study concludes that the program resulted in less intimate-partner violence, but not in less safe sex or in slowing the spread of HIV.

A few possible interpretations of this result:

• The researchers’ interpretation is that the program was responsible for reductions in violence, but that these simply didn’t translate into slowing the spread of HIV. Definitely a possibility. (If this is right, by the way, I’d call this a great program solely on the basis of its successfully reducing intimate-partner violence. That would be a great accomplishment in its own right, even if it didn’t have the hoped-for effect on the spread of HIV.)
• It’s also possible that the program had no effect, and that the observed change was a change in reported episodes of violence. Perhaps women who participated in the program came to feel more shame about reporting these episodes. (It’s also possible that the measurement error is in the other direction - that women in the program felt more pressure to report episodes, and that the fall in violence was greater than what was measured. This is the researchers’ theory, given on pg 1982.)
• And it’s possible that random fluctuations simply swamped any effects of the program itself. As mentioned above, it examined only 8 villages; and there was definitely a lot else happening in these villages over the time period in question. For example, the unspecified measure of “greater food security” had a huge rise across all villages studied, whether or not they received the program (see pg 1980). I can’t help but wonder: if this had been a more typical (less rigorous) study without a comparison group, would this increase in food security have been touted as a success of the program?

The one thing I feel fairly sure of after reading this study is that the researchers’ elaborate, multi-step theory of how loans and education can slow the spread of HIV didn’t come out looking great when all the facts were in. For every community program that publishes a study like this (and this is one of the very few I’ve seen), there are many more similar programs, with similarly involved theories of the linkages between credit, knowledge, health, empowerment, etc. that have simply never been checked in any way.

Antiretroviral treatment (ART) is one of the more well-publicized ways to help people in the developing world. The (RED) campaign puts it front and center, and the Gates Foundation places heavy emphasis on it as well. It seems at first glance like a fairly straightforward, if expensive, intervention: directly treat HIV-positive people with proven drugs to extend their lifespan and improve quality of life.

But the Copenhagen Consensus disease experts (also lead authors on the Disease Control Priorities report) make the case for caution (from pg 40 - emphasis mine):

• Poor implementation (low adherence, development of resistance, interruptions in drug supplies) is likely to lead to very limited health gains, even for individuals on therapy. (This outcome is unlike that of a weak immunization program in which health gains still exist in the fraction of the population that is immunized.) Poorly implemented antiretroviral drug delivery programs could divert substantial resources from prevention or from other high-payoff activities in the health sector. Even worse, they could lead to a false sense of complacency in affected populations: evidence from some countries suggests that treatment availability has led to riskier sexual behavior and increased HIV transmission. The injunction to “do no harm” holds particular salience.
• Unless systematic efforts are made to acquire hard knowledge about which approaches work and which do not, the likelihood exists that unsuccessful implementation efforts will be continued without the appropriate reallocation of resources to successful approaches. Learning what works will require major variations in approach and careful evaluation of effects. Failing to learn will lead to large numbers of needless deaths. Most efforts to scale up antiretroviral therapy unconscionably fail to commit the substantial resources required for evaluation of effects. Such evaluations are essential if ineffective programs are to be halted or effective ones are to receive more resources.
• Many programs rely exclusively on the cheapest possible drugs, thereby risking problems with toxicity, adherence, and drug resistance. From the outset a broader range of drug regimens needs to be tested.

An ART program needs to use the right drugs, ensure compliance, be there for the long haul, and deal with side effects (both medical and behavioral). None of these are a given, with the (RED) campaign’s beneficiaries or anyone else, until you see the evidence that the programs are working.

And ART costs can be in the range of $600 per patient treated per year. Compare with vaccinations, which are estimated as saving lives for as little as $200 apiece, have a strong track record of success, and in many ways introduce less potential for complications.

The Disease Control Priorities Report says:

The recommended treatments for malaria in areas with resistance to single drugs are combination treatments, preferably artemisinin combination therapy (ACT) (WHO 2001a, 2001b, 2003a, 2005).

But, knowing that your charity of choice runs this program is not sufficient to know that they’re improving lives. Bill Brieger at Malaria Matters points to this article in the WSJ which says:

Cures for malaria are largely designed for adults; the pills are often bitter and too big to swallow for children, who account for most of the more than one million people killed each year by the mosquito-borne disease, malaria experts say.

Bill Brieger adds:

Three challenges that are not mentioned in the article include -

• For one, when drugs are made available for free or at reduced cost only for children, there will be leakage into wider use as health workers or medicine shop keepers will provide multiple packets of the child drugs to satisfy their adult clients/customers.
• A second unmentioned challenge is the tendency to overprescribe malaria drugs, especially among adults. The answer to this is case management that includes diagnosis using a laboratory, but more likely rapid diagnostic tests, which can be used at the primary care level.
• Finally there is the issue of compliance. Artemisinin-based combination therapy generally is taken twice a day for three days. If medicine providers do not counsel clients on the need for full compliance children may swallow only a few doses and not only fail to be cured but also contribute to drug resistance.

Malaria case management is a complicated process that begins with the drug manufacturer and ends in the home. All partners along the way must be [vigilant] if children’s lives are to be saved.