The GiveWell Blog

Funding research

(updated on: ) | by Holden

At some point we’d like to investigate the idea of donating to research projects. Non-profit-motivated research is credited with many large and meaningful successes, both in medical areas (most recently, the development of a rotavirus vaccine) and in other areas (most notably, the Green Revolution).

There are serious concerns when donating in this area. For example, a recent working paper by Austan Goolsbee (h/t Greg Mankiw) argues that the additional dollar of U.S. research funding doesn’t lead to more research – only to higher salaries for the people already in the area.

One of the big challenges, it seems to me, will be coming up with ways to make good/educated guesses at which areas of research are funded to the point of diminishing returns, and which aren’t.

Key resources will be surveys such as the relatively new public survey of research & development funding for “neglected diseases” (h/t Malaria Matters), defined as diseases that predominantly affect the developing world. Data like this could allow a “rough cut” at which areas are over vs. under-funded: look at the proportion of dollars to the death toll, or DALY burden, etc.

Of course such a heuristic can’t capture the whole picture – certain diseases may be costlier to investigate, there may be more promising paths on some than on others, etc. Is there a better way to get at this question?

Two worlds

(updated on: ) | by Holden

Why are people so excited about one study of one charter school showing improved performance on math tests? (Our coverage of the study here).

It’s because in academic circles, improving academic performance is seen as an extremely thorny problem with a very long list of past failures. (See pages 1-2 of the paper for an overview.) The very strong default assumption is that an education program will fail to improve performance. To the point where a one-time, one-standard-deviation bump in math scores is considered (by David Brooks) to be a “miracle.”

But you’d never know it from the world of education philanthropy. Attend any fundraiser or read any annual report and all you’ll hear is stories of success.

There’s a similar split between two worlds in international philanthropy. Academics nearly all stress the challenges, the frustrations, and the sense that progress hasn’t matched expectations. Talk to a charity and you’ll hear “success, success, success.”

Many people are incredulous that we recommend so few charities. I can only guess that that’s because they’re coming from the world of fundraisers, where every charity is assumed to be a success. In our world, “recommended” is the exception, not the default.

Where I stand on education, my former favorite cause

(updated on: ) | by Holden

Education used to be my favorite cause. My enthusiasm waned as I saw both the cost-effectiveness of international aid and the apparent futility of education. (Elie’s 2007 post captures many of my thoughts.) The study that I’ve been blogging about today (here and here) provides a firmer grounding for our optimism about high-intensity charter schools, and challenges the idea that there aren’t good opportunities for donors in education.

However, I’m still not ready to prioritize education again, personally. One of the things that surprised me most in studying education was not just the difficulty in finding programs that could improve academic performance, but also the complete lack of rigorous evidence that education is key to later life outcomes. I would be fascinated to see a rigorous study of how the students who benefit from excellent charter schools perform later in life – in terms of income, job satisfaction, criminal records, etc. Without evidence, I’m not convinced that raising a child’s math score raises their life prospects, especially in a way that goes beyond “signaling” (i.e., allowing them to outcompete other people due to a superior credentials).

Would putting every child in America in a good school that makes sure they can do math lead to a much better society? I used to assume it would; I’m no longer so sure, and recent information doesn’t change that.

For now, I’m going to wait and see. I’d like to see the academic reaction the Fryer and Dobbie paper on the Harlem Chilren’s Zone. If others agree about the rigor and significance of its findings, I’d like to see who steps forward to continue replicating and examining this approach. The Social Innovation Fund would seem to be one strong candidate.

In the meantime, I’m going to be putting my own money into programs that are proven and replicable and don’t have enough funding – things like tuberculosis control and distribution of insecticide-treated nets.

Perhaps, at some point, I will feel that there is an education program that meets all three of these criteria as well. At that point I may start giving to it, even if it’s many times as expensive per person as developing-world aid.

Related posts:

Fryer and Dobbie on the Harlem Children’s Zone: Significance

(updated on: ) | by Holden

My last post summarized a very recent paper by Fryer and Dobbie, finding large gains for charter school students in the Harlem Children’s Zone. (You can get the study here (PDF)).

I believe that this paper is an unusually important one, for reasons that the paper itself lays out very well in the first couple of pages: a wealth of existing literature has found tiny, or zero, effects from attempts to improve educational outcomes. This is truly the first time I (and, apparently, the authors) have seen a rigorously demonstrated, very large effect on math performance for any education program at all.

This study does not show that improving educational outcomes is easy. It shows that it has been done in one case by one program.

The program in question is a charter school that is extremely selective and demanding in its teacher hiring (see pages 6-7 of the study), and involves highly extended school hours (see page 6). Other high-intensity charter schools with similar characteristics have long been suspected of achieving extraordinary results (see, for example, our analysis of the Knowledge is Power Program). This study is consistent with – but more rigorous than – existing evidence suggesting that such high-intensity charter schools can accomplish what no other educational program can.

Those who doubt the value of rigorous outcomes-based analysis should consider what it has yielded in this case. Instead of scattering support among a sea of plausible-sounding programs (each backed with vivid stories and pictures of selected children), donors – and governments – are now in a position where they can focus in on one approach far more promising than the others. They can work on scaling it up across the nation and investigate whether these encouraging effects persist, and just how far the achievement gap can be narrowed. As a donor, the choice is yours: support well-meaning approaches with dubious track records (tutoring, scholarships, summer school, extracurriculars, and more), or an approach that could be the key to huge gains in academic performance.

Reasons to be cautious

The result is exciting, but:

  • This is only one study, and the sample size isn’t huge (especially for the most rigorous randomization-based analysis). It’s a very new study – not even peer-reviewed yet – and it hasn’t had much opportunity to be critiqued. We look forward to seeing both critiques of it and (if the consensus is that it’s worth replicating) the results of replications.
  • Observed effects are primarily on math scores. Effects on reading are encouraging but smaller. Would this improved performance translate to improvements on all aspects of the achievement gap (either causally or because the higher test scores are symptomatic of a general improvement in students’ personal development)? We don’t know.
  • Just because one program worked in one place at one time doesn’t mean funding can automatically translate to more of it. Success could be a function of hard-to-replace individuals, for example. Indeed, if the consensus is that this program “works,” figuring out what about it works and how it can be extended will be an enormous challenge in itself.
  • With both David Brooks and President Obama lending their enthusiasm fairly recently, the worthy mission of replicating and evaluating this program could have all the funding it needs for the near future. Individual donors should bear this in mind.

Related posts:

Fryer and Dobbie on Harlem Children’s Zone: What they found

(updated on: ) | by Holden

The Fryer/Dobbie study on the Harlem Children’s Zone is, in my view, an extremely important work that should seriously affect how donors think about the cause of promoting equality of opportunity in the U.S. (Longtime readers of this blog know that we don’t often say something like this.) This post will simply summarize what it found, and I’ll discuss my views on the significance in another post.

Here’s the link to the paper on Roland Fryer’s website (PDF). Note that it is prominently marked “PRELIMINARY AND INCOMPLETE” and is not yet peer reviewed. All page numbers refer to this paper.

The big picture

The study tracks children in the Harlem Children’s Zone, “a 97-block area in central Harlem, New York, that combines reform-minded charter schools with a web of community services created for children from birth to college graduation that are designed to ensure the social environment outside of school is positive and supportive” (page 2).

It makes a rigorous case, using two different methodologies, that the charter school investigated in this area (Promise Academy I, 6-8 grade) had a huge effect on children’s academic performance as measured by standardized tests. The effect size was particularly large in math – enough to close the black-white achievement gap, something that no previous (rigorously demonstrated) effect has come close to. It was much smaller in English Language Arts (ELA), but still as large an effect as has been rigorously demonstrated by just about any past study. (The paper itself gives many references to such past studies on pages 1-2.)

From limited evidence, the authors also argue that other social programs in Harlem Children’s Zone did not have large effects, and were likely not the key to the school’s effects.

The charts below are taken from pages 32-33. They show the progress of scores over time for attendees (compliers) vs. non-attendees (CCM), measured in standard deviations. (The actual meaning of these lines involves some extrapolation to correct for issues of bias – discussed more below).

Some details

For the full details, read the full study (linked above). However, here are answers to some of the first questions that I usually raise with studies like these. I split them into three parts: charter school analysis I (the methodology that generated the headline results and the charts above); charter school analysis II (an alternative methodology that found similar results); other analysis (less rigorous and less conclusive analysis about other aspects of the Harlem Children’s Zone).

Charter school analysis I

The study compared lottery winners to lottery losers at the oversubscribed Promise Academy. Due to oversubscription, the two groups had been separated using a randomized lottery – implying that the only salient difference between them was randomly determined. (For more on the advantages of randomization in assessing program effects, see the Poverty Action Lab’s writeup.) 182 6-8 grade “lottery winners” were compared to 304 “lottery losers”; the groups were initially similar according to available information (see page 35).

Attrition (students’ leaving the school) was significant, as it is in many schools (see page 12). However, the analysis did not directly compare attendees to non-attendees. It first compared “lottery winners” to “lottery losers,” ignoring whether winners had actually enrolled, and found significant effects on both math and reading scores (see the top charts on pages 32-33; these are not the same as the charts above in this post). It then used statistical extrapolation (2SLS approach) to estimate the differences between those who attended and those who did not attend only because they had lost the lottery. (The second approach is what generated the charts above.)

Statistically significant and unprecedented gains were seen in math scores for all subgroups. Effects on ELA test scores were relatively large compared to past effects of educational interventions, but were not as large, and were not statistically significant except in 8th grade. (See page 38.)

A somewhat similar analysis was used for grades K-3, but the school was not significantly oversubscribed (see page 3), making the estimation messier. The study found positive but not statistically significant effects on test scores, and statistically significant effects on attendance rates (see page 41).

Added 2:29pm: the study also partially addresses the possibility of cheating:

Jacob and Levitt (2003) use an algorithm for detecting teacher cheating to show there are serious cases of teacher
or administrator cheating on high-stakes tests in four to five percent of Chicago elementary schools. While we do
not have the question-by-question data necessary to run the Jacob-Levitt algorithm, we have the results of low-
stakes interim test scores given by the charter schools for internal instruction purposes. Student performance on the
low-stakes tests is comparable to the high-stakes tests.

Charter school analysis II

The authors did an alternate analysis “tak[ing] advantage of two important features of the HCZ charter schools: (1) anyone is eligible to enroll in HCZ’s schools, but only students living inside the Zone are actively recruited by the HCZ staff; and (2) there are cohorts of children that are ineligible due to the timing of the schools’ inception and their age” (page 3). Examining tens of thousands of children, their estimation found that being eligible due to age and physically in the Harlem Children’s Zone had significant effects, over and above having either one of these characteristics (page 37).

This analysis is not as rigorous as the first method described, but still seems reasonably strong/persuasive to me. It found, again, large and statistically significant effects on math and positive but smaller/non-statistically significant effects on ELA scores for 6-8 graders (page 37). It also found large and statistically significant effects on both for grades K-3 (page 40).

Other analysis

Pages 19-21 describe less rigorous analysis of the Harlem Children’s Zone’s early childhood programs, failing to find significant effects.

Page 22 argues that the effects described above should be attributed primarily to the schools and not to the network of community services, for reasons including:

  • School attendees living outside of the Harlem Children’s Zone (and thus ineligible for many of the community services) benefited just as much as those within the Zone.
  • Lottery winners’ siblings (who became eligible for many family benefits) saw positive but small gains in test scores and attendance.

Related posts:

Harlem Children’s Zone closes achievement gap?

(updated on: ) | by Holden

Fascinating claim reported by David Brooks. The study (by Roland Fryer and Will Dobbie) doesn’t seem to be available anywhere as of this writing.

Fryer and his colleague Will Dobbie have just finished a rigorous assessment of the charter schools operated by the Harlem Children’s Zone. They compared students in these schools to students in New York City as a whole and to comparable students who entered the lottery to get into the Harlem Children’s Zone schools, but weren’t selected …. the most common education reform ideas — reducing class size, raising teacher pay, enrolling kids in Head Start — produce gains of about 0.1 or 0.2 or 0.3 standard deviations … Promise Academy produced gains of 1.3 and 1.4 standard deviations. That’s off the charts. In math, Promise Academy eliminated the achievement gap between its black students and the city average for white students.

It’s a strong claim about one of the best-reputed and -publicized charities working on the cause of equality of opportunity. We’ve been checking the Harlem Children’s Zone website (and contacting their representatives) for years, without seeing any documentation of impact – but perhaps it’s now on the way. If anyone gets a copy of this paper or knows how we can, please let us know.

Update: thanks to everyone who sent me a link to the paper. I have now read it, and my initial impression is that this is an extremely important work that should seriously affect the way a donor views this cause. I will be making at least one more post on the topic today, and possibly more than one.