Evaluating charter schools, continued

We appreciate the feedback we got on our last blog post. This post will address the substantive issues raised by commenters.

Evaluation and test scores

We aren’t seeking to settle the question of how to measure success in education. That would be biting off far more than we can chew. Instead, we have largely invited applicants to make their case in their own terms, and we aim to grant the one whose case we find the most convincing. This is an unavoidable choice for any donor; our goal is not to seek an unrealistic level of certainty, but simply to put far more time and effort into it than a typical donor can. That said, here are our thoughts on what we find compelling as a measure of success:

Ideally, we’d like to evaluate these organizations based on how well they impact the long-term, life outcomes for the students they serve – this would involve measures such as earnings, criminal record, or maintaining health insurance. But based on general conversations with highly experienced people in the fields of education-related grantmaking and research (including a professor at Harvard’s Graduate School of Education and an experienced program officer at a relevant foundation); conversations with our applicants; and the 1-2 days we spent digging through NCES, IES, and JSTOR/Google Scholar/etc., we don’t believe we’re going to be able to connect K-12 programs to these sorts of outcomes, even loosely, because this type of evaluation does not exist. We also don’t believe we’re going to be able to connect them to college performance, matriculation and graduation, etc., at least in the case of the particular applicants we’ve been discussing, because we’ve asked for this information and they have not provided it.

However, we have seen these applicants point to test scores as evidence for effectiveness, and while this measure leaves a lot to be desired, we still think it’s worth strong consideration. As many have pointed out, improved test scores could be indicative of “teaching to the test” rather than a better education, but given that we are trying to help disadvantaged students who are frequently deficient in basic math and reading skills, we’d guess that improving their ability to do well on reading and math tests more likely than not corresponds with improving their education and prospects. We would guess that improved classroom management (i.e., keeping students attentive – something we believe is a major issue, based on conversations with teachers who work in inner-city public schools), which likely indicates a better classroom environment for education, would also correspond, more likely than not, with improved test scores.

The bottom line is that while this measure leaves a lot unanswered, there are many reasons to see it as meaningful – yet the research we’ve seen does not address specific questions that we feel could be addressed with the information available. We are specifically referring to the questions of (a) whether students are systematically entering these schools with higher performance to begin with; (b) how much of the effect on test scores can plausibly be attributed to attrition. We believe both of these questions could be better (though still far from perfectly) answered through the process outlined before, i.e., examining:

  • The difference in the test scores of our applicants’ students versus students in schools in the same area.
  • The difference in the change in test scores of our applicants’ students versus students in schools in the same area – particularly for the year in which students enter the schools in question.
  • The trajectory of the change in test scores. (I.e., does it all happen in one year or is it consistent over multiple years)
  • What proportion of students apply to the lottery, and what proportion leave each year, to get a sense of how plausible these factors are in explaining any difference in test scores.

More specific questions

Michelle: thanks for your recommendations of sites to look at. These are our thoughts:

  1. We looked at ERIC several months ago for contextual research on educational evaluation, but should have revisited it for papers on these applicants specifically. We’ve now gone back and found a few papers on KIPP (including this one that looks particularly interesting), and we’ll make sure to look at all of them as we do this research. This was a very helpful suggestion, and we appreciate it.
  2. http://ies.ed.gov/ncee/ – we’ve looked before at what seem to us to be the most relevant areas in this resource. Specifically, I’ve looked at the What Works Clearinghouse: A central and trusted source of scientific evidence for what works in education and the Evaluation Studies of the National Center for Evaluation. Both areas have a great deal of information on education but none that deals directly with either our applicants or their methods.
  3. http://www.greatschools.net/ – we looked at this earlier in the year and didn’t find it to have the kind of information we’re looking for. It offers parent reviews (usually only a handful, if that); a “Great Schools rating” (which compares a school’s test scores to the the state average; and a link to the test scores available on the state websites. Is there something else here that we missed?

Matt, Andrea, Crystal:

We believe that community-driven research and grantmaking is an idea that has potential, and there are projects (such as Great Nonprofits) that are actively working on it. However, this approach carries an additional set of challenges that we don’t feel equipped to take on. Based on our own experience trying to do this work part-time, we believe it is important to get a starting point that is driven by full-time, grant-backed research. We set our own research approach and encourage those who are interested in the issues to give feedback on it; that, not asking that our process be designed from scratch by our readers, is the intent of these posts.

Comments

Evaluating charter schools, continued — 8 Comments

  1. I like the goal of focussing on longterm life changing things, even if the data doesn’t exist — it seems like it’s the sort of thing that it would be worthwhile to ask each school every time they present to you, even if they don’t have the answers to get them to start thinking about the problem in that manner.

    You’re explanation for why to focus on test scores (“it’s the only data that exists consistantly”) makes sense too — you’d have to be giving a lot of money to change what data got recorded and how. this also makes me think it will be difficult to track individual success since it seems unlikely that there’d be any individual identifiers for test scores once the data reached a public level.

  2. Have you seen this?:

    http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

    It describes program logic models, which are used by many nonprofits to attempt to frame the measurement of their effectiveness. The further out in time the outcome is, the less information is available. Schools would have to follow a large enough sample of their alumni through their lives to be able to draw any conclusions of the type you describe–an extremely expensive proposition.

    You might give grants to a set of schools to pursue such a project. That would certainly add to the collective wisdom in the way you hope.

  3. Having worked in a charter school environment my main advice would be to focus young. By the time a student is in high school and not able to read well it’s too late to avoid reading being associated with shame and failure. Difficulty appropriate reading materials are not age appropriate so it’s not possible to make a student read without shaming them.
    As for math, even arithmetic is so much less important than reading that it might as well not exist. People who can’t do the simplest arithmetic word problems in the abstract still know they are being ripped off if someone asks $100 for a pizza. Really, lots of middle class adults do find with arithmetic skills so shabby that calculating a tip is an issue.

  4. Michael Vassar said: ” By the time a student is in high school and not able to read well it’s too late to avoid reading being associated with shame and failure. Difficulty appropriate reading materials are not age appropriate so it’s not possible to make a student read without shaming them.”

    I don’t understand the logic here. Because it’s difficult on many levels, one shouldn’t attempt to teach older kids to read? I really don’t think it’s zero sum, or that donors should give up on older kids. In fact, high school kids are right at the threshold of being unreachable by the school system, by definition, so that the attempt to help then can make a tremendous difference, since it’s in a way a last chance. I think the problem of shame is interwoven throughout all of education, and that it doesn’t make sense to use it as a reason not to do something, since it is always a factor.

    As far as math goes, I think a big part of the issue isn’t just that it might be hard to do math for everyone anyway, so that therefore math isn’t important, but that if a child is to advance to college and beyond, he or she needs to be able to work up to a certain level in several subjects, including math, just to be considered for college.

  5. As always Andrea, I think that money should be spent where it will have the most benefit. The question is “elementary or high school?”, and the answer is “high school”. It doesn’t really matter that this is the last chance for a particular bunch of kids, as there will continue for some time to be kids who can benefit and not enough money to benefit all of them.

    Regarding math, not everyone goes to college or needs to. In the US a majority of those who attend don’t graduate, and many of those who don’t accumulate serious debt. A large majority of US adults lack degrees. Other developed countries have fewer college grads than the US. This is completely compatible with high standards of living.

  6. I didn’t say that one shouldnt try to figure out where the most benefit would accrue, I just didn’t think your rationale made any sense.

    Regarding not worrying about math in an intervention in a school setting, again, I’m not sure your rationale makes any sense.

    “not everyone goes to college or needs to.” Part of the point of intervening in a public school setting is to make sure that kids who have the potential to succeed in colllege have the opportunity to do so, despite influences in their lives that might make it difficult to succeed in school. Attempting to teach math is a part of this. Again, your rationale for not worrying about math doesn’t seem related to the whole concept of intevening in a school setting.

  7. You guys are still here, huh?

    And your expertise in education policy is certified how? Where did you do the doctorate?