The Career Academies initiative recently released a report on academies’ impact on students eight years after graduation. I’m fascinated by this report and this initiative because:

The Career Academies initiative rejects conventional wisdom about education.

For much of my life I’ve assumed that learning math, reading, and other “liberal arts” related skills is the key to later success in life. All of the K-12 education-focused charities we’ve examined appear to have the same unspoken assumption, stressing the importance of academic success (and generally measuring success through outcomes such as graduation rates and test scores). Last year, however, we started questioning this basic logic (for which we’ve found no empirical support).

Career Academies, while not ignoring academics, explicitly focus on preparing students for specific jobs (for example, see the Introduction of the full report recently released). And according to the evaluation, they are improving students’ earnings without improving their graduation rates or test scores (more below). Rather than assuming that the academic gap is at the heart of the achievement gap, this initiative is going straight after the latter.

The evaluation design used by Career Academies causally connects education with later life outcomes.

None of the K-12 education-focused charities we’ve examined make any attempt to examine later life outcomes, particularly earnings. Only three of them use the kind of experimental design that points strongly to effects of the program, rather than to selection bias issues. The Career Academies evaluation does both – making it the only study I know of that can plausibly discuss the effects of a particular K-12 program on the outcomes we really care about (not test scores, but earnings).

Randomized lotteries were used to assign the limited number of slots at Career Academies, and the lotteried-out students were compared to lotteried-in students a full eight years after graduation. We’re still waiting for the “technical” companion to the evaluation to be published so we can fully examine the methodology, but according to Evidence-Based Programs, the study had important strengths such as low attrition and an intention-to-treat approach that imply that any differences between the two groups (lotteried-in and lotteried-out) can be attributed to the effect of the schools themselves. It found that lotteried-in students report over $200/mo more in earnings (see Pg 13 of the full report), and that Career Academies students report higher earnings whether they’re classified as high-, medium-, or low-risk students (Pg 26).

This benefit came despite no apparent impact (see Pgs 28-32) on traditional measures of educational progress, including test scores, graduation rates, and college enrollment/completion. (Also note that these measures show enormous selection bias; a less rigorous, more typical study would have erroneously concluded that Career Academies do affect academic performance.)

I wish we’d discovered Career Academies earlier, and checked it out as thoroughly as our K-12 applicants. Rather than chasing small (or perhaps illusory) improvements in test scores and/or graduation rates, in the hopes that classic unproven assumptions about the importance of a high school education are correct, we might have funded an intervention with a truly different approach and a truly thorough commitment to making sure it’s changing lives, not just grades.

The National Academy Foundation focuses on the Career Academies approach. We haven’t done a thorough investigation of it, but you might want to check it out.