We appreciate the feedback we got on our last blog post. This post will address the substantive issues raised by commenters.

Evaluation and test scores

We aren’t seeking to settle the question of how to measure success in education. That would be biting off far more than we can chew. Instead, we have largely invited applicants to make their case in their own terms, and we aim to grant the one whose case we find the most convincing. This is an unavoidable choice for any donor; our goal is not to seek an unrealistic level of certainty, but simply to put far more time and effort into it than a typical donor can. That said, here are our thoughts on what we find compelling as a measure of success:

Ideally, we’d like to evaluate these organizations based on how well they impact the long-term, life outcomes for the students they serve – this would involve measures such as earnings, criminal record, or maintaining health insurance. But based on general conversations with highly experienced people in the fields of education-related grantmaking and research (including a professor at Harvard’s Graduate School of Education and an experienced program officer at a relevant foundation); conversations with our applicants; and the 1-2 days we spent digging through NCES, IES, and JSTOR/Google Scholar/etc., we don’t believe we’re going to be able to connect K-12 programs to these sorts of outcomes, even loosely, because this type of evaluation does not exist. We also don’t believe we’re going to be able to connect them to college performance, matriculation and graduation, etc., at least in the case of the particular applicants we’ve been discussing, because we’ve asked for this information and they have not provided it.

However, we have seen these applicants point to test scores as evidence for effectiveness, and while this measure leaves a lot to be desired, we still think it’s worth strong consideration. As many have pointed out, improved test scores could be indicative of “teaching to the test” rather than a better education, but given that we are trying to help disadvantaged students who are frequently deficient in basic math and reading skills, we’d guess that improving their ability to do well on reading and math tests more likely than not corresponds with improving their education and prospects. We would guess that improved classroom management (i.e., keeping students attentive – something we believe is a major issue, based on conversations with teachers who work in inner-city public schools), which likely indicates a better classroom environment for education, would also correspond, more likely than not, with improved test scores.

The bottom line is that while this measure leaves a lot unanswered, there are many reasons to see it as meaningful – yet the research we’ve seen does not address specific questions that we feel could be addressed with the information available. We are specifically referring to the questions of (a) whether students are systematically entering these schools with higher performance to begin with; (b) how much of the effect on test scores can plausibly be attributed to attrition. We believe both of these questions could be better (though still far from perfectly) answered through the process outlined before, i.e., examining:

• The difference in the test scores of our applicants’ students versus students in schools in the same area.
• The difference in the change in test scores of our applicants’ students versus students in schools in the same area – particularly for the year in which students enter the schools in question.
• The trajectory of the change in test scores. (I.e., does it all happen in one year or is it consistent over multiple years)
• What proportion of students apply to the lottery, and what proportion leave each year, to get a sense of how plausible these factors are in explaining any difference in test scores.

More specific questions

Michelle: thanks for your recommendations of sites to look at. These are our thoughts:

1. We looked at ERIC several months ago for contextual research on educational evaluation, but should have revisited it for papers on these applicants specifically. We’ve now gone back and found a few papers on KIPP (including this one that looks particularly interesting), and we’ll make sure to look at all of them as we do this research. This was a very helpful suggestion, and we appreciate it.
2. http://ies.ed.gov/ncee/ – we’ve looked before at what seem to us to be the most relevant areas in this resource. Specifically, I’ve looked at the What Works Clearinghouse: A central and trusted source of scientific evidence for what works in education and the Evaluation Studies of the National Center for Evaluation. Both areas have a great deal of information on education but none that deals directly with either our applicants or their methods.
3. http://www.greatschools.net/ – we looked at this earlier in the year and didn’t find it to have the kind of information we’re looking for. It offers parent reviews (usually only a handful, if that); a “Great Schools rating” (which compares a school’s test scores to the the state average; and a link to the test scores available on the state websites. Is there something else here that we missed?

Matt, Andrea, Crystal:

We believe that community-driven research and grantmaking is an idea that has potential, and there are projects (such as Great Nonprofits) that are actively working on it. However, this approach carries an additional set of challenges that we don’t feel equipped to take on. Based on our own experience trying to do this work part-time, we believe it is important to get a starting point that is driven by full-time, grant-backed research. We set our own research approach and encourage those who are interested in the issues to give feedback on it; that, not asking that our process be designed from scratch by our readers, is the intent of these posts.