I think it is always interesting and meaningful when a well-designed study fails to find the impact it was looking for. It implies that something doesn’t work the way we thought, or as well as we had thought. It isn’t the last word on the impact of Head Start, and the context you provide is important as well.

Our results also complement the findings of studies on the long-term impacts of other early childhood interventions (reviewed in Almond and Currie 2010). For example, the Perry preschool program randomized 123 children into a control group and an intensive pre-school treatment group. Schweinhart, Barnes, and Weikhart (1993), Schweinhart et al. (2005), and Heckman et al. (2010a, 2010b) show that the Perry preschool program had extremely large impacts on earnings and other adult outcomes despite relatively rapid fade-out of test score impacts. Heckman et al. (2010c) show that the Perry intervention also improved non-cognitive skills and argue that this mechanism accounts for much of the long-term impact. Campbell et al. (2002) show that the Abecedarian project, which randomized 111 children into intensive early childhood programs from infancy to age 5, led to lasting improvements in education and other outcomes in early adulthood. At a larger scale, several studies have shown that the Head Start program leads to improvement in a variety of adult outcomes despite fade-out on test scores (e.g., Currie and Thomas 1995, Garces, Thomas, and Currie 2002, Ludwig and Miller 2007, Deming 2009). The results reported here are the first experimental evidence on the long-term impacts of a scalable intervention in a large sample with minimal attrition. In particular, we show that a better classroom environment from ages 5-8 can have substantial long-term benefits even without intervention at earlier ages. This result is consistent with the findings of Card and Krueger (1992), who show that better educational inputs have substantial long-term payoffs using state-by-cohort variation.

Although the study you’re discussing is of incredibly high quality, I would want significantly more information about the longer-term effects before accepting that Head Start’s impact is as limited as the study suggests.

One of the challenges of evaluation is finding appropriate measures. I feel that social programs are currently often oversold, to the point where people expect to see many sorts of impacts that are probably unrealistic - and focus studies on looking for these impacts. This phenomenon leaves us with a lot of disappointing results for programs that may nonetheless be successful, and may partially explain the Stainless Steel Law of Evaluation. We don’t conduct our own studies, so we can’t really do anything about this; we have to take what academia provides. What we can say is that results are “disappointing” relative to what was apparently hoped for.

You probably saw the study (PDF) that the New York Times reported on last month about how big the impacts of kindergarten classes (and teachers) may be on eventual earnings. One of the really interesting things that you can see on pages 48 & 49 of the linked PDF is that the effect of kindergarten class quality on grade-level test scores appears to fade out rapidly, but manifests itself quite strongly in eventual earnings. The authors hypothesize that this is because the tests stop assessing some skills that may ultimately effect income, but which may have played a role in kindergarten tests. (As they always say: all I really need to know, I learned in kindergarten…)

Could that phenomenon (of the tests ceasing to examine the relevant variables) also explain some of the fade-out observed in the Head Start study? It would seem to cohere with results of some other early childhood interventions which have limited to moderate academic effects, but extraordinary longer-term results.

I only glanced at the Head Start study, but it looks like they’re using the same assessments over the entire data collection, so changes in what is being assessed probably could not explain the fade-out effects. That said, do you think it is possible that the same assessment administered to four- and seven-year olds might tell you different things? More specifically, that an elementary reading assessment of a four-year old would tell you about something like attention span or effort, which might impact income decades later, whereas with a seven-year old the same assessment might say more about the quality of reading instruction?