We ask you, as a donor, to turn down some great pitches – “Your interest-free loan will help this person escape poverty forever,” “You can give a cow to a poor family for Christmas,” etc. – and give instead to charities that aren’t terribly good at storytelling. Why?

It comes down to this. We think that most of those stories are just that – stories. (For more, see our summary of recent posts on “big-name” charities, which we feel are representative of the full set of charities we’ve reviewed.) But if you give to one of our top charities, you really can save (or dramatically change) a human life.

It hasn’t been easy to find charities that we can honestly say this about. That’s what our process is built around and where most of our energy goes. This week we’ve blogged about the best we’ve found, VillageReach and Stop Tuberculosis Partnership. There is plenty of room for doubt even with them, but overall we think there is a strong case - even for the skeptic - that your donation to them can save a life.

What do we mean when we say “save a life?”

By “you can save a life,” we don’t mean anything as simple, concrete, or easy to grasp as the stories charities usually tell.

• Your gift can’t literally be linked to an individual. It will help an organization that, all things considered, is achieving a lot of impact for what it spends.
• If you must know what “your” dollars are doing, it’s likely that they’ll be sitting in reserves to ensure financial stability, or enabling a slightly larger travel budget for evaluators, or something similarly unexciting.
• It’s even highly possible that your donations will be wasted, and that the charity you give to – even the best you can find – will fail. We don’t think there are true guarantees in aid.
• Even if these charities are succeeding, it’s very likely that your donation won’t ultimately result in the charity doing anything differently. It’s pretty hard to think about how $1000, by itself, could really change anything about Stop TB Partnership’s plans for next year.
• Yet donations add up. 50-100 of these donations could mean a significantly larger grant, more people getting tuberculosis treatment … and that could mean families staying intact instead of being struck by sudden death.

The truth is that it takes a lot of abstraction and analytical thinking to really think about how your donation saves a life. The life you can save is an “expected” life (”expected” in the sense of probabilistic expected value) - it isn’t a person we can point to or show you a picture of. More than with typical charities, you have to use your imagination. But more than with typical charities, your impact is real.

With opportunity comes responsibility

In The Life You Can Save (which prominently features GiveWell and which we have reviewed), Peter Singer writes:

By donating a relatively small amount of money, you could save a child’s life … we all spend money on things we don’t really need, whether on drinks, meals out, clothing, movies, concerts, vacations, new cars, or house renovation. Is it possible that by choosing to spend your money on such things rather than contributing to an aid agency, you are leaving a child to die, a child you could have saved? (pg 5)

Our corollary: is it possible that you are leaving a child to die when you choose to donate to a charity with a “feel-good” story rather than a charity with a great case for real impact?

It is true that, as our critics often point out, a charity can be impactful without being demonstrably impactful. But when one charity proves itself and another leaves you guessing, it seems clear to us which one offers the “better bet” – and more “expected lives saved” – given the information available. When you have the option of giving to an outstanding charity that demonstrably can save a life, how do you justify giving to a charity whose true impact is essentially a big question mark?

I’ll leave this blog’s last words for 2009 to Natalie, a relatively recent GiveWell hire (she has been working full-time on research since July).

Sometimes I’m almost tempted to give to a charity I know less about. I’ve been over VillageReach and I’ve seen how complex the situation is and how many questions there are. If I gave to some charity I know nothing about, I could just think about the story they tell and feel good and not have these nagging doubts. But I’m not going to do that – in the end it’s more important to me that I really make a difference.

Aid Watch:

“Local people may be the experts, but for outsiders deciding where their donations can do the most good, getting access to local knowledge and acting on it appropriately requires real-time feedback loops that most aid projects lack.

Over a little more than a year, GlobalGiving combined staff visits, formal evaluation, third-party observer reports called visitor postcards, and internet feedback from local community members to create a nuanced, evolving picture of a community-based youth organization in Western Kenya that had received $8,019 from 193 individual donors through the GlobalGiving website.

Initially, youth in Kisumu were happy with the organization. Among other things, the founder used the money to fund travel and equipment for the local youth soccer team. But the first tip-off that something was going wrong came when a former soccer player complained through GlobalGiving’s online feedback form that “currently the co-ordinator is evil minded and corrupt.” The view that the founder had begun stealing donations and was stifling dissent among his members was expanded upon by other community members, visitors to the project, and a professional evaluator.

In the end, a splinter group broke off and started a new sports organization, and the community shifted their support to the new group. Reflecting the local consensus, GlobalGiving removed the discredited organization from its website.” (Emphasis mine)

Aid Watch stresses the “new way to evaluate a project” angle on this story, and we think it’s a good angle. But another angle is that most aid projects don’t receive this level of scrutiny, and the project that was put under this particular microscope - more as a way of testing the microscope than because there were concerns about the project - turned out discredited.

This is a story that I feel should affect your default assumption about whether an aid project is working.

The comments on Aid Watch’s post are also worth reading. One problem with the “funding a project at a time from many different organizations” approach is that it isn’t clear what one does with evaluation and feedback, when it’s available. Knowing how a project went is certainly better than not knowing, but the ultimate goal is to translate knowledge of what’s working into improved performance.

That’s an argument for focusing on the organization rather than the project level. Organizations can be given incentives to learn from their mistakes and improve their projects. Funding tiny organizations for one-off projects, it isn’t clear how to impose any kind of accountability.

If you don’t have evidence one way or the other, should you assume an aid’s projects benefits are reaching the poorest?

We think it’s fair to assume the people with the most need are the people with the least power. We’d also guess that, in general, the people with the most power are best positioned to get anything valuable (training, materials, loans, or whatever else) a charity is subsidizing.

There are ways a charity can get around this dynamic, such as:

• Working in an area where everyone is in need. We believe that such areas exist, but just because an area has low average incomes, high disease rates, etc. doesn’t mean it has no relatively privileged and powerful members.
• Running programs that are only appealing to those in need. Some health programs work this way (it’s hard to be treated for tuberculosis unless you have tuberculosis). Microfinance may work this way when interest rates are competitive with (not highly subsidized relative to) alternative sources of credit.
• Carefully targeting those in need. We should expect this to be very difficult. Charity is inherently about coming into a community from the outside; targeting those in need will generally mean trying to outmaneuver at least some locals. The more locals genuinely share the charity’s mission, the better, but how is the charity to know which it’s dealing with?

Unfortunately (as usual), there isn’t much information out there about how often charities actually succeed in targeting those in need. In conducting our grant application process, we’ve found that systematic assessment of this question is relatively rare. But there is certainly room for concern, as shown by a World Bank review that we’ve quoted before:

The frequent tendency for participatory projects to be dominated if not captured by local elites is highlighted by several case studies. Katz and Sara (1997), in a global review of water projects, find numerous cases of project benefits being appropriated by community leaders and little attempt to include households at any stage … even well trained staff are not always effective in overcoming entrenched norms of exclusion. In a study of community forestry projects in India and Nepal that worked reasonably well, Agarwal (2001) reports that women were systematically excluded from the participatory process because of their weak bargaining power. Rao and Ibanez (2003) find that in the participatory projects in their Jamaican case study, wealthier and better networked individuals dominated decision making. In a similar case-based evaluation of social funds in Jamaica, Malawi, Nicaragua, and Zambia, the World Bank (2002) Operations Evaluation Department concludes that the process was dominated by “prime movers.”

Abraham and Platteau (2004) present evidence on community participation processes in Sub-Saharan Africa based largely on anecdotal evidence from their work in community-based development and on secondary sources. They argue that rural African communities are often dominated by dictatorial leaders who can shape the participation process to benefit themselves because of the poor flow of information. (40-41)

The New York Times recently profiled Chess in the Schools:

The Chess-in-the-Schools program has sought to foster analytical skills on the theory that these will help students succeed academically. The group teaches 20,000 children a year and calculates that it has taught 425,000 children since 1986. Children gather to learn the game at the group’s headquarters in Manhattan.

It seems like 20 years and 425,000 children is quite a lot of investment in the “theory that [chess] will help students succeed academically.” The Times feature provides a calming justification for the investment: “Chess helps promote intellectual growth and has been shown to improve academic performance.” Let’s look at the evidence for this claim.

The study we found

An early-1990s study looks at achievement test scores of chess-playing students over two years at District 9 in the Bronx. It observes that (a) the overall average reading score improved among chessplayers by about 5 percentile points, but didn’t improve among the set of remaining District 9 students; (b) 15 of 22 second-year participants improved their reading scores by some amount, while only 491 of 1118 non-participants in the district - and 245 of 655 non-participants with high reading scores, improved.

This study is riddled with major problems:

• The numbers the researchers choose to compare seem arbitrary and possibly cherry-picked. Why do the researchers look at the “percentage who improved” among second-year chessplayers but not for both years? Why do they compare the second-year students to “high-performing nonparticipants,” but not give the same comparison when looking at all students?
• The problem of selection bias is unusually obvious here. They’re comparing kids who volunteered to play chess against those who didn’t. Think of the chess club members at your school, and ask yourself if they would have been just like all the other kids had chess club not been offered. There’s no reason to think these two groups of kids are otherwise similar or would be expected to respond similarly to school.
• This is a study of somewhere between 22 and 53 students at a single district in the early 1990s. Even if the study were highly rigorous, it would still be a long way from “proof that chess helps promote intellectual growth.”

The studies we couldn’t find

The Chess-in-the-Schools website states:

In 1991 and 1996, Stuart M. Margulies, Ph.D., a noted educational psychologist, conducted two studies examining the effects of chess on children’s reading scores. The studies demonstrated that students who participated in the chess program showed improved scores on standardized tests. The gains were even greater among children with low or average initial scores. Children who were in the non-chess playing control group showed no gains.

Another study in 1999, measured the impact of chess on the emotional intelligence of fifth graders. The results of the study were striking. The overall success rate in handling real life situations with emotional intelligence was 91.4% for the children who participated in the Chess-in-the-Schools program. In contrast, those who were not involved with the chess program had an average overall success rate of only 64.4%.

We’re guessing that the study we’re looking at is an update of the 1991 study since it references no previous studies and discusses results from 1991 and 1992. We can’t find the other studies anywhere. Chess-in-the-Schools provides neither links nor citations.

Even in the best-case scenario, it’s apparently been at least a decade since the last test of the Chess-in-the-Schools model.

“Chess helps promote intellectual growth and has been shown to improve academic performance?”

In researching charities, one of the more discouraging things we’ve learned is how little support it takes for a statement like “Chess helps promote intellectual growth and has been shown to improve academic performance” to be repeated by charities, donors, and even the media.

As far as we can tell, Chess-in-the-Schools is not a demonstrated success story. It’s just been promoted and scaled up like one.

I recently attended a seminar with the fascinating Seth Godin and heard an interesting anecdote about VisionSpring:

I could see that every single person who came to this meeting had enough money [$3] … to buy a pair of new reading glasses. And I could tell from how old they were that they were qualified and I knew what they did for a living so I knew that this would pay for itself in two weeks, three weeks, certainly in 20 years it’s going have a huge return on investment. So they get the demonstration, they take the eye test, they see that they need glasses, they take the sample glasses off, they walk over to the table where there are 8 kinds of glasses to choose from … all carefully wrapped in plastic … and 40% of the people bought a pair of glasses. 60% of the people left. And I’ve thought about this about a million times … If they knew how great the glasses would be, if they could overcome the momentum they had and the desire to keep the money … there’s no question they would have bought a pair of glasses. Maybe for $6 or $9 or $12, they could afford it, they needed it, they were in the right place at the right time, and yet the transfer didn’t occur.

To Mr. Godin, it seemed obvious that the customers should be buying glasses, and that they were being held back by “momentum.” He proposed a change:

Instead of giving the person the eye test, taking off the glasses and having them go over … and now make the decision do you care enough about yourself to buy those, instead, give the guy the eye test, and say those are your glasses, you owe me $3. Now the person has to make a new decision, which is better, giving up the $3 and keeping on what I have or going to the trouble of taking them off and reminding myself that I don’t deserve to see? And when you do that it turns out that the close rate goes up 30%.

To be sure, it sounds like an impressive improvement for a simple change. And yet a 30% improvement on a 40% close rate still leaves about half the people not spending the glasses - now under circumstances that arguably make it pretty difficult to turn them down.

Is it possible that Mr. Godin – and the charity he was observing – have simply overestimated the demand for what they’re doing? Underestimated the extent to which imperfect substitutes may be available? It appears that Village Phone did exactly this in at least one case.

It’s easy to be sure that your product is great and that it’s needed. Yet if you’re wrong, and you have donors subsidize it, you may essentially be giving out cash in a less efficient, less empowering way.

Mr. Godin analogized the situation to fundraising:

You’re sitting in class and the person next to you … is coughing … and you take out a container of Fisherman’s Friend menthol lozenges and you offer it to her and she puts it in her mouth and the coughing goes away and everyone is happy … and yet you’ve been on those calls to raise money from a donor, who apparently has money to donate … and you hand them the equivalent of a Fisherman’s Friend, “look at this program, we’ve been working really hard on it, it’s really important, we need your money” and they say “I need to get back to you” and you know what that means …

To me, there’s a major problem with the assumption that a charity’s program is at effective at solving complex social problems as a Fisherman’s Friend is at soothing throats.

Confidence in your product is great, but it can be misplaced. When charities let assumptions like “Everyone here needs glasses” and “Our program is reliable as a cough drop” go unexamined, they’re going to be left scratching their heads at results-oriented donors.

Ever since I heard about the Grameen Foundation’s Village Phone program, I’ve been optimistic. The program involves helping people in remote villages run pay-for-use cellphone services: they get their cellphone, and a loan to buy it, via Grameen, then charge other villagers to use it. It’s an approach to fighting poverty that’s (a) relatively new; (b) using a product that hasn’t been available for a long time but seems clearly useful to anyone doing business in remote areas; (c) utilizing a “franchise model” where people in the villages take a stake in the product.

It was near the top of my “Probably helping people, even though we don’t yet have systematic evaluation yet” list. Now Chris Blattman points to a discouraging evaluation that found “absolutely no impact of the phones on trading activity or availability of goods in local markets” and very small (non-significant) impacts on profits and measures of well-being (school enrollment, consumption of meat, etc.)

This bottom-line result does not, by itself, mean the program “doesn’t work.” It could work very differently in different contexts (discussed below), and there are some possible issues with the paper (which is very recent, and is not a randomized controlled trial). But one thing I like about the study is that it doesn’t just discuss impact - it examines many aspects of the program, and exposes assumptions that may otherwise have gone unquestioned.

Assumption 1: the phones are in high demand and operators easily cover their costs. In fact, usage of the phones was around 4 hours a month, or 8 minutes a day (pg 19). As a result, profits from the phones were not enough to keep up with loan payments (pg 19). Grameen reportedly has responded by changing loan and franchise terms (pg 30). Tuvugane (pg 5), a less sophisticated phone product that was already common in the villages, may have been good enough for most purposes.

Assumption 2: farmers who use the phones benefit from better pricing power. Even though farmers with access to the phones became much more likely to arrange their own transport to market, there was no apparent effect on the prices they received for their goods, possibly due to established relationships with buyers (pg 16).

Assumption 3: if someone chooses to become a cellphone operator, they’re going to benefit from it. In fact, there was a very strange pattern in the businesses of people who became phone operators. Their hours worked rose significantly both for their new phone business and for their already-existing businesses, but their profits and wages paid did not rise (pgs 17-18). A possible explanation is that operators wanted to be available for cellphone users and so stayed at their workplaces longer, but that the extra hours didn’t translate into extra profits. In any case, it’s a pattern that doesn’t seem encouraging, and seems to deserve further investigation.

Bottom line: a product that was supposed to be helpful and in high demand arguably ended up as a bad investment for the franchise operators. This doesn’t mean it shouldn’t have been tried, or that it shouldn’t be tried in the future. But it points to the importance of testing assumptions empirically, rather than scaling up a program as widely as possible based on an appealing story.