The GiveWell Blog

Pneumonia Day is another General Health Day

| by Holden

On World Pneumonia Day, it’s natural for a donor to ask, “Where can I give to fight pneumonia?”

But the actual plan to fight pneumonia consists of the following: breastfeeding promotion, vaccination, and treatment in communities, health centers, and hospitals.

All of these are general health initiatives that touch on a wide variety of health problems. So it’s not surprising to see that almost none of the World Pneumonia Day Coalition members are organizations devoted to “pneumonia”; they are vaccine organizations, general charities, and even malaria and tuberculosis organizations.

If you are a “traditional” donor, we hope World Pneumonia Day succeeds in making you support these extremely worthwhile initiatives, by playing to your need for breaking news, grand announcements, and shocking facts that you didn’t know/remember last month.

If you are a results-oriented donor, you don’t need World Pneumonia Day. Just support great global health charities.

Why can’t you make the sale?

(updated on: ) | by Holden

I recently attended a seminar with the fascinating Seth Godin and heard an interesting anecdote about VisionSpring:

I could see that every single person who came to this meeting had enough money [$3] … to buy a pair of new reading glasses. And I could tell from how old they were that they were qualified and I knew what they did for a living so I knew that this would pay for itself in two weeks, three weeks, certainly in 20 years it’s going have a huge return on investment. So they get the demonstration, they take the eye test, they see that they need glasses, they take the sample glasses off, they walk over to the table where there are 8 kinds of glasses to choose from … all carefully wrapped in plastic … and 40% of the people bought a pair of glasses. 60% of the people left. And I’ve thought about this about a million times … If they knew how great the glasses would be, if they could overcome the momentum they had and the desire to keep the money … there’s no question they would have bought a pair of glasses. Maybe for $6 or $9 or $12, they could afford it, they needed it, they were in the right place at the right time, and yet the transfer didn’t occur.

To Mr. Godin, it seemed obvious that the customers should be buying glasses, and that they were being held back by “momentum.” He proposed a change:

Instead of giving the person the eye test, taking off the glasses and having them go over … and now make the decision do you care enough about yourself to buy those, instead, give the guy the eye test, and say those are your glasses, you owe me $3. Now the person has to make a new decision, which is better, giving up the $3 and keeping on what I have or going to the trouble of taking them off and reminding myself that I don’t deserve to see? And when you do that it turns out that the close rate goes up 30%.

To be sure, it sounds like an impressive improvement for a simple change. And yet a 30% improvement on a 40% close rate still leaves about half the people not spending the glasses – now under circumstances that arguably make it pretty difficult to turn them down.

Is it possible that Mr. Godin – and the charity he was observing – have simply overestimated the demand for what they’re doing? Underestimated the extent to which imperfect substitutes may be available? It appears that Village Phone did exactly this in at least one case.

It’s easy to be sure that your product is great and that it’s needed. Yet if you’re wrong, and you have donors subsidize it, you may essentially be giving out cash in a less efficient, less empowering way.

Mr. Godin analogized the situation to fundraising:

You’re sitting in class and the person next to you … is coughing … and you take out a container of Fisherman’s Friend menthol lozenges and you offer it to her and she puts it in her mouth and the coughing goes away and everyone is happy … and yet you’ve been on those calls to raise money from a donor, who apparently has money to donate … and you hand them the equivalent of a Fisherman’s Friend, “look at this program, we’ve been working really hard on it, it’s really important, we need your money” and they say “I need to get back to you” and you know what that means …

To me, there’s a major problem with the assumption that a charity’s program is at effective at solving complex social problems as a Fisherman’s Friend is at soothing throats.

Confidence in your product is great, but it can be misplaced. When charities let assumptions like “Everyone here needs glasses” and “Our program is reliable as a cough drop” go unexamined, they’re going to be left scratching their heads at results-oriented donors.

The Gates Foundation’s agriculture program: Experimenting or floundering?

(updated on: ) | by Holden

Here’s what we know about the Gates Foundation’s agriculture program:

  • Gates believes it’s suggestive that “apart from a few states and small, oil-rich countries, no country has managed a rapid rise from poverty without increasing agricultural productivity. In the poorest countries, agriculture employs a majority of the people.”
  • This isn’t a new argument or an undisputed one. See Peter Timmer on Green Revolution “optimists” vs. “pessimists”.
  • Gates’s approach is “comprehensive,” targets “no single, simple solution”, and includes farmer training/support, irrigation initiatives, market access initiatives, and funding of agricultural research with a focus on gender empowerment.
  • This isn’t a new approach or a historically successful one. The World Bank has focused on essentially the same set of interventions recently, with unclear results, and the previous “holistic” approach of “Integrated Rural Development” is widely considered to have failed. Details at our overview of agriculture aid.

In other words, the Gates Foundation approach – as described – appears to be neither a continuation of things that have worked before nor a fundamentally new approach to the problem. So what might be different this time around?

Lots of things. Better technology could make all the difference; so could a greater degree of commitment. And one way in which the Gates Foundation could really distinguish itself from past efforts would be by doing a superior job learning about what works and what doesn’t – past initiatives have suffered from poor evaluation and very little accessible information about how things have really worked out.

The Gates Foundation’s progress reports so far are extremely preliminary, looking at “inputs” such as “number of farmers organized into groups”. We find these measures wholly appropriate given how early it is in the initiative; and yet, we’ve seen too many programs that still haven’t moved beyond these measures even after claiming “success” and asking individuals to donate and help them scale up.

If the Gates Foundation moves to more rigorous and outcome-focused evaluation over time, we might learn more about what works and what doesn’t, and the “impatient optimism” could turn out to be justified. If not, the Gates Foundation will be making a very expensive gamble with very little information about its odds.

Gates Foundation on agriculture funding: Where are the facts?

(updated on: ) | by Holden

The Gates Foundation states that “funders have sharply cut their international aid to agricultural development over the past few decades.” It is implied that this is a major reason for the failure to see a “Green Revolution in Africa.”

We have been unable to locate support for this claim.

Using data from OECD – the most reliable source of official aid flows as far as we know – we graphed the proportion of (disbursed) aid to Africa that was classified in the “agriculture” sector since the 1970s. It’s possible that the numbers are artificially depressed for early years due to less standardized reporting, but we see no trend of the type the Gates Foundation describes.

In addition, we previously looked at funding on some of the main vehicles credited with the original “Green Revolution” and found no substantial drop since the 1970s.

It’s frustrating that the Gates Foundation doesn’t provide a source for its claim. In general, the material on its website is at a very broad level and does not make it possible for people curious about its underlying reasoning to drill deeper. That means that any criticism or examination of its work has to be either very superficial or done offline (i.e., in direct communication with the Foundation, an opportunity few people seem to get easily or often).

Added 10/30/09: here’s our source data

Village Phone: Another great story under the microscope

(updated on: ) | by Holden

Ever since I heard about the Grameen Foundation’s Village Phone program, I’ve been optimistic. The program involves helping people in remote villages run pay-for-use cellphone services: they get their cellphone, and a loan to buy it, via Grameen, then charge other villagers to use it. It’s an approach to fighting poverty that’s (a) relatively new; (b) using a product that hasn’t been available for a long time but seems clearly useful to anyone doing business in remote areas; (c) utilizing a “franchise model” where people in the villages take a stake in the product.

It was near the top of my “Probably helping people, even though we don’t yet have systematic evaluation yet” list. Now Chris Blattman points to a discouraging evaluation that found “absolutely no impact of the phones on trading activity or availability of goods in local markets” and very small (non-significant) impacts on profits and measures of well-being (school enrollment, consumption of meat, etc.)

This bottom-line result does not, by itself, mean the program “doesn’t work.” It could work very differently in different contexts (discussed below), and there are some possible issues with the paper (which is very recent, and is not a randomized controlled trial). But one thing I like about the study is that it doesn’t just discuss impact – it examines many aspects of the program, and exposes assumptions that may otherwise have gone unquestioned.

Assumption 1: the phones are in high demand and operators easily cover their costs. In fact, usage of the phones was around 4 hours a month, or 8 minutes a day (pg 19). As a result, profits from the phones were not enough to keep up with loan payments (pg 19). Grameen reportedly has responded by changing loan and franchise terms (pg 30). Tuvugane (pg 5), a less sophisticated phone product that was already common in the villages, may have been good enough for most purposes.

Assumption 2: farmers who use the phones benefit from better pricing power. Even though farmers with access to the phones became much more likely to arrange their own transport to market, there was no apparent effect on the prices they received for their goods, possibly due to established relationships with buyers (pg 16).

Assumption 3: if someone chooses to become a cellphone operator, they’re going to benefit from it. In fact, there was a very strange pattern in the businesses of people who became phone operators. Their hours worked rose significantly both for their new phone business and for their already-existing businesses, but their profits and wages paid did not rise (pgs 17-18). A possible explanation is that operators wanted to be available for cellphone users and so stayed at their workplaces longer, but that the extra hours didn’t translate into extra profits. In any case, it’s a pattern that doesn’t seem encouraging, and seems to deserve further investigation.

Bottom line: a product that was supposed to be helpful and in high demand arguably ended up as a bad investment for the franchise operators. This doesn’t mean it shouldn’t have been tried, or that it shouldn’t be tried in the future. But it points to the importance of testing assumptions empirically, rather than scaling up a program as widely as possible based on an appealing story.

Helping farmers is harder than you’ve heard

(updated on: ) | by Holden

Imagine that a charity is able to teach a farmer some basic, useful things about farming (like “crop rotation, dip irrigation and the planting of trees that enrich over worked soil” or “disease-resistant cassava replication, distribution and sale; crop diversification; soil conservation; and expanding market opportunities”). Such simple knowledge could last the farmer forever and be far more useful – especially for the cost – than cash or loans. It’s an often-sold story, and an appealing one.

What charities don’t tell you about “improved farming techniques and technology” is just how long the aid world has been trying to spread them, and how much it has struggled. The basic challenges:

Can agriculture programs reach enough farmers? The right farmers?

A 2006 World Bank paper examines the long history of “agricultural extension” programs and is frank about their problems. For traditional programs, it states that

The cost of reaching large, geographically dispersed and remote smallholder farmers is high, particularly given high levels of illiteracy, limited access to mass media, and high transport costs. Farming systems often entail several crops, livestock, and even within given geographical area, there are variations in soil, elevation, microclimate and farmers’ capabilities and access to resources. With such a large and diversified clientele, only a small fraction of farmers can be served directly (face-to-face) by extension, and agents tend to focus on the larger, better resourced and more innovative farmers. This reduces the potential for farmer-to-farmer diffusion. (Emphasis ours)

The “Training & Visit” model attempted to address these issues through a strong, clear set of hierarchies and responsibilities (see pgs 11-14), but its substantially higher costs – coupled with the fact that, as with previous programs, impact was hard to see – led to its essentially universal abandonment (see pgs 14-15 and pgs 22-23).

When World Vision or Save the Children speaks of spreading improved practices, is it using a “T&V” style intensive-but-costly approach, or a lighter touch that could fail to reach enough (and the right) farmers? It isn’t clear.

Do charities even know what to teach and what to change?

Another general problem cited by the World Bank paper is that “Weak accountability (linked to the inability to attribute impact) is reflected in low-quality and repetitive advice given to farmers, and in diminished effort to interact with farmers, and to learn from their experience.” (Emphasis ours.) In other words, those giving advice may not actually be giving the right advice.

It is hard to find honest and thorough descriptions of how such projects have actually played out in the past, but a couple of striking failure stories should make it clear just how badly outsiders can misjudge what farmers need to learn:

  • The DrumNet program in Kenya aimed, successfully, to transition farmers from growing “local crops” (i.e., crops for local/personal consumption) to growing “export crops” (i.e., crops to be sold on the export market). However, a year after the project evaluation was completed, the firm that had been buying the “export crops” stopped due to European regulations, leading to “the collapse of Drumnet as farmers were forced to undersell to middlemen, leaving sometimes a harvest of unsellable crops and thus defaulting on their loans.” (Details at this paper published on the Poverty Action Lab site (PDF).)
  • A development program in Lesotho aimed to help local people with crop and livestock management, as well as building roads so they could access markets. However, few of the people in the region were farmers, and conditions were not good for farming. Harsh weather destroyed pilot crop projects, and the roads allowed in competitors who drove the existing local farmers out of business. (From pgs 193-4 of White Man’s Burden)

These aren’t cases of minor missteps – they’re cases where those giving aid did not perceive essential and fundamental aspects of the local economy. That doesn’t mean they were incompetent – it means that understanding a local economy well enough to give truly useful advice may not be easy.

The long and murky history of agricultural assistance

Agricultural programs in Africa have struggled to produce tangible results, both at the micro level (little evidence about how programs have gone) and at the macro level (disappointing progress in Africa-wide crop yields over time).

A variety of approaches have been tried, including the “holistic” approach of simultaneously addressing health, transportation, credit, and agricultural knowledge. This approach was referred to as “Integrated Rural Development” in the 1970s and 1980s and appears to be acknowledged as a failure, although the basic idea behind it may be making a comeback in the “holistic” approach of the Millennium Villages Project and other large charities.

Details at our writeup on agriculture-focused aid.

Bottom line for donors: agricultural technology is not like medicine

Agriculture aid is often presented as a matter of extending the reach of proven technologies and methods. However, the track record of such programs is simply nothing like that of health programs, which often have track records including multiple highly rigorous studies and large-scale, demonstrable successes.

We feel that the burden of proof on agriculture programs is high, but outcomes tracking of any kind is extremely rare. The evaluations that are available tend to raise many concerns about whether results are “cherry-picked” and whether results even point to improved lives.

We recommend that donors be extremely wary of charities working heavily in this area, no matter how good their intentions. We have not identified any that we can have confidence in.