Here’s a grant idea we’d probably pursue if we had the funds to do so. I’d be interested in what other grantmakers think of it. I believe enormous good could be done by offering grants to charities that can prove their programs don’t work.

Proving a program’s ineffectiveness is difficult and expensive – just as much as proving its effectiveness. Few charities have done either. As a result, the world knows extremely little about what programs do and don’t work, and thus about how to change people’s lives for the better.

You should consider this a serious problem unless you think that nearly all charitable programs are effective. (If you think this, you should consider this list of duds).

Of course, one solution is to reward charities that are effective and can demonstrate it. That’s the approach we generally take. But there are problems with the incentives this approach (by itself) gives to unproven charities.

We’d like an unproven charity to examine its programs rigorously, and get our recommendation if the results turn out positive. But what if the programs it’s running turn out not to work (i.e., change lives)? Then the charity will have spent money and time to weaken its own case. Something of a scary proposition – and a reason to be less than evenhanded in conducting evaluations.

But what if a foundation said the following? “If you can really prove that your program isn’t working – not just that it’s underfunded or has room for improvement, but that it fails to change lives when carried out correctly – that’s valuable. That improves our collective knowledge of what works, and demonstrate a true commitment to your mission, not just your activities.

“For a charity that can prove its programs aren’t working, we’ll provide you with the funding to redesign what you’re doing. If you have the right mission and the wrong tactics – and the guts to admit it – we’ll help you change those tactics, so you can accomplish the goal (saving lives, promoting equality of opportunity, etc.) that you really care about.”

If there were enough funding along these lines, carefully examining effectiveness – with no preconceptions or manipulation, just an honest desire for better knowledge – would be win-win for a charity. As it should be.

In 2017, the organization 80,000 Hours conducted a project to update the findings in this post. Their work is here.

This is a guest post from David Anderson, Assistant Director at the Coalition for Evidence-Based Policy, the group responsible for the Evidenced-based Programs website. Mr. Anderson’s responsibilities include reviewing studies of effectiveness and looking for proven programs. He’s worked at the Coalition since 2004. The views expressed below are solely those of the writer.

The holiday season often heightens our desire for “feel good” stories. Rigorous research has shown that it can fulfill this desire by identifying, in a scientifically rigorous way, a few social programs and services that are capable of producing important, positive effects on people’s lives.

For example, the Nurse Family Partnership – a nurse home visitation program for women who are poor, mostly single, and pregnant with their first child (recommended by GiveWell here) – has been shown in three rigorous randomized studies to produce long-term, 40-70% reductions in child abuse and neglect, as well as important improvements in cognitive and academic outcomes for the most at-risk children served by the program. The very existence of research-proven programs like the Nurse Family Partnership, suggests that a concerted effort to build the number of these programs through rigorous research, and to spur their widespread use, could fundamentally improve the lives of millions of people.

However, what’s not often recognized is how rare—and therefore, valuable—such examples of proven effectiveness are. Their scarcity stems from two main factors: 1) the vast majority of social programs and services have not yet been rigorously evaluated, and 2) of those that have been rigorously evaluated, most (perhaps 75% or more), including those backed by expert opinion and less-rigorous studies, turn out to produce small or no effects, and, in some cases negative effects. These are factors we’ve identified through our reviews of hundreds of studies that federal agencies, Congress, and philanthropic organizations have asked us to look at.

The following are just a few illustrative examples of this general pattern of many good ideas turning out not to work when tested in a rigorous way:

• 21st Century Community Learning Centers—a rigorous randomized study of after school programs at 26 elementary schools, funded by the U.S. Department of Education, found that, on average, they had no effect on students’ academic achievement and had negative effects on their behavior (i.e. increased rates of school suspensions and other disciplinary problems compared to control group students). (See note 1 below)
• Many leading educational software products—a rigorous randomized study of 16 leading educational software products for teaching reading and math – including many award-winning products – found, on average, no difference in reading or math achievement between students who used them in their classrooms, and those who were taught through usual methods. (See note 2 below)
• Even Start—a rigorous randomized evaluation of 18 Even Start family literacy programs funded by the U.S. Department of Education found that, on average, children and adults served by the programs scored no higher on reading tests than their control group counterparts. (See note 3 below)
• New Chance Demonstration—a rigorous randomized evaluation of 16 schools and organizations funded by the U.S. Department of Labor to provide comprehensive case management to teenage mothers, designed to improve their employment outcomes, found that, on average, these programs had no effect on their employment or earnings. (See note 4 below)

Importantly, these findings do not mean that all after-school programs, educational software, family literacy programs, and case management services are ineffective– just that many widely used approaches in these areas don’t work, and additional research is needed to identify those that do. While such findings are disheartening, they illustrate the importance of targeting government funding, as well as individual donations, on the relatively few programs and practices that have been shown in rigorous evaluations to be highly effective. Doing so will increase the likelihood that public and private dollars are truly going to help improve people’s lives in important ways—and that’s something to feel good about this holiday season.

References:

1. James-Burdumy et al. “When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program Final Report.” U.S. Department of Education/Institute of Education Sciences National Center for Evaluation and Regional Assistance. April 2005.
2. Dynarski, et. al. “Effectiveness of Reading and Mathematics Software Products: Findings From the First Student Cohort: Report to Congress.” U.S. Department of Education/Institute of Education Sciences National Center for Evaluation and Regional Assistance. March 2007.
3. St. Pierre et. al. “Third National Even Start Evaluation: Program Impacts and Implications for Improvement.” U.S. Department of Education’s Planning and Evaluation Service. 2003.
4. Quint et al. “New Chance: Final Report on a Comprehensive Program for Young Mothers in Poverty and Their Children.” MDRC. 1997

Nathaniel Whittemore’s Social Entrepreneurship Blog asks bloggers for “one thing you need to know before you donate to charity this holiday season.” My answer: you need to know that your favorite social program might just not work.

This isn’t a warning against fraud or inefficiency (though both of those are important too). It’s a warning against programs that just don’t change people’s lives in the way we hope – even if they seem to make perfect sense, and even if they’re carried out perfectly.

The first $17,000 I ever donated (personally) was to programs that I now believe don’t work. During my years in the finance industry, I gave to the best organizations I could find for improving education (jr. high and high school) in NYC.

I considered education my favorite cause. I assumed that equality of schooling was the key to equality of opportunity. I didn’t have the time or the energy to question this assumption. I now believe this assumption is badly wrong, for reasons that are outlined here.

I wish I could take that money back: de-fund the “small schools” and extracurricular activities I supported (both of these are programs I now know to have very questionable, if not negative, track records) and instead fund programs for early childhood (where I believe inequality of opportunity really begins) or international aid (where it’s far more drastic).

I wish that money had gone to organizations that I really believe are changing lives in a significant and lasting way, but it didn’t. Please don’t make my mistake.

There are great-sounding programs out there, based on one theory or another of what the roots of poverty and opportunity are. When put under the microscope, many of these great-sounding programs just don’t work, most likely because they simply didn’t take the right approach to the complicated problems they’re trying to solve. Many more of these programs are unexamined and unproven.

Charities that will put your money into proven ways of helping people are the exception, not the rule. They’re not necessarily easy to find. They’re not necessarily the same ones that knock on your door and get your friends excited. This season, with or without GiveWell’s help, I hope you find one.

A study published in the New England Journal of Medicine reports strong results from a clinical trial testing the efficacy of a malaria vaccine. The double-blind, randomized trial found a significant impact on malaria incidence among the group receiving the malaria vaccine.

The full study is available here.

The summary results:

• 894 children were randomly assigned to receive the either the malaria vaccine or hte control vaccine.
• Among the children who completed the study according to the protocol, 32 of 402 (8%) who received the malaria vaccine developed clinical malaria and 66 of 407 (16%) in the control group did.
• Results were similar when all 894 children were included in the intention-to-treat analysis.
• Results were similar in both the trial locations (Tanzania and Kenya).

(h/t Chris Blattman)

I’ll be on NPR Marketplace today, talking about why we think you can do more good by putting time into researching your giving than into volunteering. Local air times/stations here. I expect the audio will be online afterward (at which point I’ll edit it into this post).

The first question I have for any charity is, “What do you do?” Not “What are you trying to accomplish?” (example: “Fight AIDS”) but “What activities are you carrying out and where?” (example: “School-based education programs emphasizing protection in Mozambique”).

It’s a simple question, and an obviously important one: as the Disease Control Priorities Report makes quite clear, some program types have strong track records and some have none (or negative) track records.

But at least in the area of international aid, “What do you do?” can be a difficult question to answer.

Example: Africare

Take Africare for example. Go to the homepage and you’ll see press releases from scattered programs. Click Programs and you’ll see goals/categories (“HIV/AIDS,” “Health,” etc.) Clicking Learn more for HIV/AIDS takes you to an overview of the problem of HIV/AIDS. Now an “Africare Programs” tab appears at the top, giving you the following list:

• HIV prevention
• HIV/AIDS counseling and testing
• Treatment, palliative care and other support for people living with HIV, AIDS and TB/HIV
• Support for AIDS orphans

The 2nd and 3rd of these are fairly clear (though these activities do come in different “flavors” – for example, ART treatment with and without supplementary feeding). The 4th eventually links to another page with more detail. But “HIV prevention” can mean many things: abstinence education, prevention education, peer-based/school-based/media-based programs, condom distribution, circumcision and more. And, we still don’t know where any of this takes place.

In Africare’s case, the answer to my question turns out to be on pages 19-22 of the Annual Report. Africare runs a dizzying array of programs, each sponsored by a particular set of donors. To me it appears to be almost a “subcontractor” organization, carrying out what its major donors want rather than pursuing any particular strategy of its own. (More on this idea in a future post.)

No help from the Form 990

The IRS Form 990 is supposed to be the source for basic public information on a charity. But while it tells you plenty about revenues, expenses, and assets, it rarely answers the “What do you do?” question. Charities are asked only to provide only a brief, overarching mission statement.

As a result, you can’t find out what Africare does from its GuideStar report (free registration required) or its Charity Navigator report.

Creating a database

We wish there were a database that told us what activities charities carry out and where. Such a database wouldn’t, by itself, answer the “Where should I donate?” question we’re eventually trying to get to. But it would be a huge help, and it doesn’t exist right now.

So we’re trying to create it. We’ve found that you can usually uncover what a charity does with enough digging (as in the above case). Now we’re trying to capture this information for all or nearly all international aid organizations that are (a) relatively large (b) US-registered public charities.

This is the main work we’re looking for a prospective new hire to do.