The GiveWell Blog

Career Academies: An unconventional approach to education

The Career Academies initiative recently released a report on academies’ impact on students eight years after graduation. I’m fascinated by this report and this initiative because:

The Career Academies initiative rejects conventional wisdom about education.

For much of my life I’ve assumed that learning math, reading, and other “liberal arts” related skills is the key to later success in life. All of the K-12 education-focused charities we’ve examined appear to have the same unspoken assumption, stressing the importance of academic success (and generally measuring success through outcomes such as graduation rates and test scores). Last year, however, we started questioning this basic logic (for which we’ve found no empirical support).

Career Academies, while not ignoring academics, explicitly focus on preparing students for specific jobs (for example, see the Introduction of the full report recently released). And according to the evaluation, they are improving students’ earnings without improving their graduation rates or test scores (more below). Rather than assuming that the academic gap is at the heart of the achievement gap, this initiative is going straight after the latter.

The evaluation design used by Career Academies causally connects education with later life outcomes.

None of the K-12 education-focused charities we’ve examined make any attempt to examine later life outcomes, particularly earnings. Only three of them use the kind of experimental design that points strongly to effects of the program, rather than to selection bias issues. The Career Academies evaluation does both – making it the only study I know of that can plausibly discuss the effects of a particular K-12 program on the outcomes we really care about (not test scores, but earnings).

Randomized lotteries were used to assign the limited number of slots at Career Academies, and the lotteried-out students were compared to lotteried-in students a full eight years after graduation. We’re still waiting for the “technical” companion to the evaluation to be published so we can fully examine the methodology, but according to Evidence-Based Programs, the study had important strengths such as low attrition and an intention-to-treat approach that imply that any differences between the two groups (lotteried-in and lotteried-out) can be attributed to the effect of the schools themselves. It found that lotteried-in students report over $200/mo more in earnings (see Pg 13 of the full report), and that Career Academies students report higher earnings whether they’re classified as high-, medium-, or low-risk students (Pg 26).

This benefit came despite no apparent impact (see Pgs 28-32) on traditional measures of educational progress, including test scores, graduation rates, and college enrollment/completion. (Also note that these measures show enormous selection bias; a less rigorous, more typical study would have erroneously concluded that Career Academies do affect academic performance.)

I wish we’d discovered Career Academies earlier, and checked it out as thoroughly as our K-12 applicants. Rather than chasing small (or perhaps illusory) improvements in test scores and/or graduation rates, in the hopes that classic unproven assumptions about the importance of a high school education are correct, we might have funded an intervention with a truly different approach and a truly thorough commitment to making sure it’s changing lives, not just grades.

The National Academy Foundation focuses on the Career Academies approach. We haven’t done a thorough investigation of it, but you might want to check it out.

Comments

  • michael vassar on July 17, 2008 at 10:46 am said:

    This is GREAT information. Thanks!

  • rob s on July 19, 2008 at 1:54 pm said:

    i agree w/the hunch the basic assumptions commonly made are highly flawed when it comes to education. are there not more outfits like MDRC who did this study and is partnered with Ford Foundation that Givewell could rely on?

  • Holden on July 21, 2008 at 1:46 pm said:

    Rob: there is a lot of independent research out there. Going through it and identifying/learning from what’s useful is a big task; it’s something that we’re working on now, and there are hopefully more posts like this one coming up.

    Our favorite three sources for this sort of research are discussed in this blog post.

Comments are closed.