Would you rather hear about a good charity or a bad one?

When I’m explaining GiveWell to someone, there often comes a moment where his/her eyes suddenly light up, and s/he says something like, “So you bust the bad guys, eh? Can you tell me about a really bad one you’ve nailed?” (Paraphrased.)

This sort of person is usually pretty disappointed if and when they look at our actual reviews. “I don’t get it – you said you didn’t grant this organization because you didn’t have enough evidence to assess them. So you don’t KNOW that they’re bad …”

Finding bad charities is a fundamentally different endeavor from finding good ones. In some ways, it’s more fun and more exciting to find bad ones. Scandals are juicy; qualified statements that an organization appears to be improving lives, though many factors remain unexplored, are less so. “Good vs. evil” makes headlines and produces adrenaline in a way that “Proven vs. unproven” doesn’t.

But if you’re looking to accomplish as much good as possible with your donation, and you’re looking for an organization that you can be confident is changing lives for the better, I submit that you’re much better off focusing your energies on the few charities that might be able to convincingly document their effectiveness. Evaluating well-documented charities is more than enough work. Trying to nail down the effects of charities that don’t have strong self-documentation is an enormous undertaking, one that I think is worth your time only if you have a personal connection or other strong reason to believe you’re already dealing with an exceptional organization. Spending any time on organizations that don’t stand out in any way – and bothering to make distinctions between “bad” and “worst” – doesn’t seem like a good use of time at all. The question is, are you trying to make a good story or make a donation?

From 2004-2006, I gave all my donations to organizations focused on helping inner-city youth (particularly academically). Equality of opportunity was my favorite cause, and I assumed (without having time to really look into it) that inequality stemmed from the gap in quality between different grade schools. I now believe that this assumption was badly wrong, and that as a result, my donations were mistargeted.

The most surprising thing I’ve learned about the achievement gap between black and white / low-income and high-income students is how early in life the gap is present. Every source I’ve looked at is consistent in this regard: children from different socioeconomic and ethnic backgrounds have large, systematic differences in academic performance in kindergarten, and these differences grow only slightly as children get older. In other words, most of the systematic academic inequality we observe is present by age 5. There is a good deal of literature on this subject, but the paper I’d recommend for starting to take a look is “Understanding Trends in the Black-White Achievement Gaps during the First Years of School” by Murnane, Willet, Bub, and McCartney (Brookings-Wharton Papers on Urban Affairs, 2006).

This is a relatively simple observation, but it completely changes the way I think about promoting equality of opportunity in the U.S.

• It makes me much more interested in interventions that focus on early childhood, the period during which most of the “achievement gap” appears.
• It makes me much more skeptical of the idea that equalizing children’s schools will equalize their educations, something that once seemed obvious to me. It makes me much less optimistic about what one can accomplish with “small schools” (a Gates-backed initiative that has produced disappointing results), private-school scholarships, etc. (The Murnane paper also references research on disappointing results from programs in this category.)
• It makes me especially skeptical of low-intensity grade-school or high-school interventions such as after-school tutoring. I find it very possible that a few hours of extra help a week just aren’t enough to make a dent in deep-rooted disadvantages.

I think it’s interesting that this extremely basic, fundamental, and important fact about the achievement gap – how much of the gap is present by age 5 – has not come up in any of our conversations with (or applications from) charities themselves. That includes both education charities and child-care charities. It seems to me that most development and fundraising professionals are focused on reinforcing and serving donors’ existing assumptions; if you want to challenge your assumptions to get the best understanding possible, you have to look elsewhere.

A new study sponsored by several major foundations (Gates, Packard, Hewlett, Irvine, and Robert Wood Johnson) found that among “engaged”* Americans, only:

• 43% can name a foundation on their first try
• 15% can cite an example of a foundation’s impact in their community
• 11% can cite an example of a foundation’s impact on an issue they care about

Individuals – who don’t have access to information about how well charitable programs are working – donate over $220 billion dollars to charity every year. Foundations retain expert staff to evaluate programs and make grant decisions. If foundations want to increase their relevance to individual citizens (and those citizens’ awareness of them), one good start might be addressing this information gap: using their expertise to help donors make more informed giving decisions.

* U.S. adults aged 18 and older who have held a leadership, committee or board level role in a group or organization working on a community or social issue within the past year.

The Brookings Institution is hosting a conference this week called “What works in development?” including an interesting paper by Simon Johnson (International Monetary Fund) and Peter Boone (London School of Economics) titled, “Do Health Interventions Work? Which and in What Sense?”

Johnson and Boone review the existing literature and conclude that there is very little knowledge about the most effective methods for reducing child mortality in the developing world, and that without improved knowledge, aid organizations may fail to reduce child mortality as much as they hope.

Knowledge is limited in the following ways:

• We know what works clinically, but know far less about the most effective ways to fully implement an intervention.

  This rings true to me and makes me think of bednets. We know that insecticide treated bednets, when used properly and consistently prevent deaths from malaria (at least in the short term), but we don’t know the most effective way to ensure proper use, a critical component of the intervention. Evidence for the effectiveness of bednets comes from aid experience in very specific contexts (e.g., the way in which the nets are distributed, the education level of those who receive them, etc) which means that the distributing bednets may not be as effective when implemented in a context different from that of the initial evaluations.

• When we have multiple, proven interventions, we generally don’t know which to implement where or how they’d work as a package.

  Keeping the point above in mind, there’s strong evidence that both insecticide treated bednets and artemisinin-based therapies reduce child mortality when implemented properly. However, little is known about how they work together (as a package) or which situations are best suited to one or the other. (It doesn’t make sense to implement both everywhere because a) the more the treatment is used the more quickly resistant strands of malaria will likely develop and b) the cost of implementing both everywhere will obviously exceed an approach that implements only what is necessary).

• Evaluations of interventions’ effectiveness often stop at measuring reduction in incidence as opposed to total mortality.

  Often, evaluations of interventions focus on an intervention’s effect on disease incidence (e.g., the reduction in cases of diarrhea caused by building improved water and sanitation infrastructure). This is a problem because many of the causes of death in the developing world are interrelated – i.e., one problem increases the likelihood of death from another. UNICEF estimates that malnutrition is a contributing factor to 50% of child deaths (from malria, diarrhea, etc.), and the WHO finds that measles contributes deaths from pneumonia and diarrhea. Because of these interrelationships, evaluations that only asses an intervention’s effect on disease incidence may not accurately identify the effect on mortality.

  This problem is illustrated in a recent paper cited by Johnson and Boone that finds that while water and sanitation projects reduce incidence of diarrhea, they have a minimal impact on child mortality. Johnson and Boone hypothesize that:

  It seems plausible that the much wider coverage of water and sanitation today, along with the advent of vaccines and treatments for the main causes of death from infectious disease, mean that further improvements in water and sanitation are no longer necessary or very significant to eliminate remaining deaths (pg 21).

Johnson and Boone have their own view on the best approach: targeting parental knowledge rather than distribution of materials. They observe that:

• Many interventions are extremely inexpensive (e.g., Oral rehydration therapy costs $.10/packet and malaria treatment costs $.50 cents/dose – Pg 17), and are not beyond the means of many people in the developing-world.
• There is good evidence that parents are not very knowledgeable about health (Pg 25), and parents’ education is highly correlated with child mortality (Pg 23).

It’s a plausible hypothesis, but could easily be flawed, as Johnson and Boone point out themselves. For example, it’s possible that the observed correlation between parental education and child health is a simple consequence of the fact that more educated parents also tend to be wealthier, and that wealth is in fact the primary factor here.

Knowing that their hypothesis could be right or wrong, Johnson and Boone have set out to test it. Working with Effective Intervention, a UK-based charity, they’re planning to implement a series of randomized controlled trials of comprehensive aid programs focusing on a) educating parents and b) providing access to necessary health products. In some areas, they’ll also include education for children as part of the intervention, planning to follow the children at least 10 years after the completion of the trial. Eventually, they plan to run trials in 600 villages in Africa and India covering 500,000 children. They say it will take three years for the first findings.

This is the first we’ve heard of Effective Intervention, but they are taking exactly the approach we identify with most: starting with a systematic review of what we do know, pinpointing what it is we want to know next, and then focusing on producing that knowledge rather than on scaling up a program with unknown effectiveness. We’re looking forward to their results.