The Fryer/Dobbie study on the Harlem Children’s Zone is, in my view, an extremely important work that should seriously affect how donors think about the cause of promoting equality of opportunity in the U.S. (Longtime readers of this blog know that we don’t often say something like this.) This post will simply summarize what it found, and I’ll discuss my views on the significance in another post.
Here’s the link to the paper on Roland Fryer’s website (PDF). Note that it is prominently marked “PRELIMINARY AND INCOMPLETE” and is not yet peer reviewed. All page numbers refer to this paper.
The big picture
The study tracks children in the Harlem Children’s Zone, “a 97-block area in central Harlem, New York, that combines reform-minded charter schools with a web of community services created for children from birth to college graduation that are designed to ensure the social environment outside of school is positive and supportive” (page 2).
It makes a rigorous case, using two different methodologies, that the charter school investigated in this area (Promise Academy I, 6-8 grade) had a huge effect on children’s academic performance as measured by standardized tests. The effect size was particularly large in math – enough to close the black-white achievement gap, something that no previous (rigorously demonstrated) effect has come close to. It was much smaller in English Language Arts (ELA), but still as large an effect as has been rigorously demonstrated by just about any past study. (The paper itself gives many references to such past studies on pages 1-2.)
From limited evidence, the authors also argue that other social programs in Harlem Children’s Zone did not have large effects, and were likely not the key to the school’s effects.
The charts below are taken from pages 32-33. They show the progress of scores over time for attendees (compliers) vs. non-attendees (CCM), measured in standard deviations. (The actual meaning of these lines involves some extrapolation to correct for issues of bias – discussed more below).
For the full details, read the full study (linked above). However, here are answers to some of the first questions that I usually raise with studies like these. I split them into three parts: charter school analysis I (the methodology that generated the headline results and the charts above); charter school analysis II (an alternative methodology that found similar results); other analysis (less rigorous and less conclusive analysis about other aspects of the Harlem Children’s Zone).
Charter school analysis I
The study compared lottery winners to lottery losers at the oversubscribed Promise Academy. Due to oversubscription, the two groups had been separated using a randomized lottery – implying that the only salient difference between them was randomly determined. (For more on the advantages of randomization in assessing program effects, see the Poverty Action Lab’s writeup.) 182 6-8 grade “lottery winners” were compared to 304 “lottery losers”; the groups were initially similar according to available information (see page 35).
Attrition (students’ leaving the school) was significant, as it is in many schools (see page 12). However, the analysis did not directly compare attendees to non-attendees. It first compared “lottery winners” to “lottery losers,” ignoring whether winners had actually enrolled, and found significant effects on both math and reading scores (see the top charts on pages 32-33; these are not the same as the charts above in this post). It then used statistical extrapolation (2SLS approach) to estimate the differences between those who attended and those who did not attend only because they had lost the lottery. (The second approach is what generated the charts above.)
Statistically significant and unprecedented gains were seen in math scores for all subgroups. Effects on ELA test scores were relatively large compared to past effects of educational interventions, but were not as large, and were not statistically significant except in 8th grade. (See page 38.)
A somewhat similar analysis was used for grades K-3, but the school was not significantly oversubscribed (see page 3), making the estimation messier. The study found positive but not statistically significant effects on test scores, and statistically significant effects on attendance rates (see page 41).
Added 2:29pm: the study also partially addresses the possibility of cheating:
Jacob and Levitt (2003) use an algorithm for detecting teacher cheating to show there are serious cases of teacher
or administrator cheating on high-stakes tests in four to five percent of Chicago elementary schools. While we do
not have the question-by-question data necessary to run the Jacob-Levitt algorithm, we have the results of low-
stakes interim test scores given by the charter schools for internal instruction purposes. Student performance on the
low-stakes tests is comparable to the high-stakes tests.
Charter school analysis II
The authors did an alternate analysis “tak[ing] advantage of two important features of the HCZ charter schools: (1) anyone is eligible to enroll in HCZ’s schools, but only students living inside the Zone are actively recruited by the HCZ staff; and (2) there are cohorts of children that are ineligible due to the timing of the schools’ inception and their age” (page 3). Examining tens of thousands of children, their estimation found that being eligible due to age and physically in the Harlem Children’s Zone had significant effects, over and above having either one of these characteristics (page 37).
This analysis is not as rigorous as the first method described, but still seems reasonably strong/persuasive to me. It found, again, large and statistically significant effects on math and positive but smaller/non-statistically significant effects on ELA scores for 6-8 graders (page 37). It also found large and statistically significant effects on both for grades K-3 (page 40).
Other analysis
Pages 19-21 describe less rigorous analysis of the Harlem Children’s Zone’s early childhood programs, failing to find significant effects.
Page 22 argues that the effects described above should be attributed primarily to the schools and not to the network of community services, for reasons including:
- School attendees living outside of the Harlem Children’s Zone (and thus ineligible for many of the community services) benefited just as much as those within the Zone.
- Lottery winners’ siblings (who became eligible for many family benefits) saw positive but small gains in test scores and attendance.
Related posts: