We granted Population Services International in our Saving Lives in Africa cause, focusing largely on their condom and bednet distribution and promotion programs. Soon after we cut the check, I got into a conversation with someone who questioned whether bednets are effective at all. He pointed me to a paper, “Combating malaria morbidity and mortality by reducing transmission,” which questions whether bednets are likely to have a long-term impact on malaria mortality.

Bednets reduce the number of bites that people receive from malaria-carrying mosquitoes, which in turn reduces incidence of malaria. This has two effects: (a) it reduces malaria mortality; but (b) it also reduces acquired immunity to malaria. The authors of this paper found, empirically, that areas that had lower rates of bites had similar rates of malaria mortality but a different age distribution of deaths. In places with high bite-rates, malaria mortality was restricted to children; in places with low bite-rates, adults died as well. The authors hypothesize that bednet programs may only shift the age at which people die from malaria, and have no impact on the total number of deaths from malaria.

This doesn’t mean bednets are useless. First of all, the analysis in the paper isn’t general enough to cover every case – there could be population configurations such that protecting people during their weakest years is a net benefit. There could be regions with high enough bite-rates that nets merely reduce the number of bites without affecting immunity. Also, even if total morbidity and mortality don’t change, there is definitely value in people living longer before they die. But, if it’s true that reducing mosquito bites can matter much less over the long term than over the short term, this is important when weighing the value of bednets. This paper really highlights what systematic analysis and measuring can do – and what you can miss by skimping on them.

At the moment, we’re still working on our education cause; bednets are something we’re thinking about, not devoting our time to. But if we revisit the cause of aid to Africa next year, we plan to incorporate this research into our analysis, and possibly change our recommendations. We don’t expect our decisions to be perfect; they’re merely our “best bet” given what we know. Enabling anyone to provide feedback on our grant decisions, which refines (or contradicts) our previous understanding, is precisely why we think a public discussion of what charities do and how well it works is necessary.

Measuring the effect of aid on people’s lives can be difficult, and may never be perfect – but we believe that it can be done, and has been done, both rigorously and practically. Examples can be found on three sites devoted to conducting and/or promoting rigorous evaluation of social programs: Poverty Action Lab, Innovators for Poverty Action, and the Coalition for Evidence-based Policy.

These sites focus on examining and promoting particular sorts of programs (both in policy and charity), rather than on recommendations for donors; we’re working on determining how much of their information can be used to help with a donation decision. As it stands, though, all three provide good and plentiful examples of of how evaluations can be performed that are practical, rigorous, and ultimately capable of advancing our knowledge of what is likely to work. If you know of more sources of such evaluations, please share.

Developing world

The Abdul Latif Jameel Poverty Action Lab and Innovations for Poverty Action are both devoted to researching interventions aimed at alleviating developing-world poverty.

Both focus explicitly on conducting randomized controlled trials – studies that, in a nutshell, incorporate a “lottery” aspect into choosing a program’s clients, and compare those who were randomly chosen to participate to those who were randomly chosen not to. Differences between the two groups can often be attributed with reasonable confidence to the program itself, without many of the concerns over bias that can cloud the results of other kinds of studies.

Both of these organizations make their publications freely available, directly from their website. The rigor and availability of these studies makes it possible for a casual user to learn a great deal about what is likely to improve people’s lives in the developing world (for example, J-PAL provides a good deal of evidence for the positive potential of deworming programs).

U.S.

We’ve known about the above resources for a while, but only recently learned of the Coalition for Evidence-Based Policy, which has collected an impressive set of information on what works in developed-world causes such as child care, education, and employment. Like the resources above, this website focuses on studies using a randomized controlled trial approach.

Many of these studies provide direct evidence of particular charities’ effectiveness, and some of the others are useful in setting a general approach (for example, two of the studies they list – which we found through other methods – are cited in our earlier writeup on early childhood care, and influenced the way we think about this issue).

We plan to look closely at the studies on this website; we believe it has provided a valuable service in collecting some of the more rigorous evidence on what works, and that we’ll be able to use it to learn a great deal about how to accomplish as much good as possible with charitable donations. We’ll be sharing our developing thoughts as we go through its materials.

We appreciate the feedback we got on our last blog post. This post will address the substantive issues raised by commenters.

Evaluation and test scores

We aren’t seeking to settle the question of how to measure success in education. That would be biting off far more than we can chew. Instead, we have largely invited applicants to make their case in their own terms, and we aim to grant the one whose case we find the most convincing. This is an unavoidable choice for any donor; our goal is not to seek an unrealistic level of certainty, but simply to put far more time and effort into it than a typical donor can. That said, here are our thoughts on what we find compelling as a measure of success:

Ideally, we’d like to evaluate these organizations based on how well they impact the long-term, life outcomes for the students they serve – this would involve measures such as earnings, criminal record, or maintaining health insurance. But based on general conversations with highly experienced people in the fields of education-related grantmaking and research (including a professor at Harvard’s Graduate School of Education and an experienced program officer at a relevant foundation); conversations with our applicants; and the 1-2 days we spent digging through NCES, IES, and JSTOR/Google Scholar/etc., we don’t believe we’re going to be able to connect K-12 programs to these sorts of outcomes, even loosely, because this type of evaluation does not exist. We also don’t believe we’re going to be able to connect them to college performance, matriculation and graduation, etc., at least in the case of the particular applicants we’ve been discussing, because we’ve asked for this information and they have not provided it.

However, we have seen these applicants point to test scores as evidence for effectiveness, and while this measure leaves a lot to be desired, we still think it’s worth strong consideration. As many have pointed out, improved test scores could be indicative of “teaching to the test” rather than a better education, but given that we are trying to help disadvantaged students who are frequently deficient in basic math and reading skills, we’d guess that improving their ability to do well on reading and math tests more likely than not corresponds with improving their education and prospects. We would guess that improved classroom management (i.e., keeping students attentive – something we believe is a major issue, based on conversations with teachers who work in inner-city public schools), which likely indicates a better classroom environment for education, would also correspond, more likely than not, with improved test scores.

The bottom line is that while this measure leaves a lot unanswered, there are many reasons to see it as meaningful – yet the research we’ve seen does not address specific questions that we feel could be addressed with the information available. We are specifically referring to the questions of (a) whether students are systematically entering these schools with higher performance to begin with; (b) how much of the effect on test scores can plausibly be attributed to attrition. We believe both of these questions could be better (though still far from perfectly) answered through the process outlined before, i.e., examining:

• The difference in the test scores of our applicants’ students versus students in schools in the same area.
• The difference in the change in test scores of our applicants’ students versus students in schools in the same area – particularly for the year in which students enter the schools in question.
• The trajectory of the change in test scores. (I.e., does it all happen in one year or is it consistent over multiple years)
• What proportion of students apply to the lottery, and what proportion leave each year, to get a sense of how plausible these factors are in explaining any difference in test scores.

More specific questions

Michelle: thanks for your recommendations of sites to look at. These are our thoughts:

1. We looked at ERIC several months ago for contextual research on educational evaluation, but should have revisited it for papers on these applicants specifically. We’ve now gone back and found a few papers on KIPP (including this one that looks particularly interesting), and we’ll make sure to look at all of them as we do this research. This was a very helpful suggestion, and we appreciate it.
2. http://ies.ed.gov/ncee/ – we’ve looked before at what seem to us to be the most relevant areas in this resource. Specifically, I’ve looked at the What Works Clearinghouse: A central and trusted source of scientific evidence for what works in education and the Evaluation Studies of the National Center for Evaluation. Both areas have a great deal of information on education but none that deals directly with either our applicants or their methods.
3. http://www.greatschools.net/ – we looked at this earlier in the year and didn’t find it to have the kind of information we’re looking for. It offers parent reviews (usually only a handful, if that); a “Great Schools rating” (which compares a school’s test scores to the the state average; and a link to the test scores available on the state websites. Is there something else here that we missed?

Matt, Andrea, Crystal:

We believe that community-driven research and grantmaking is an idea that has potential, and there are projects (such as Great Nonprofits) that are actively working on it. However, this approach carries an additional set of challenges that we don’t feel equipped to take on. Based on our own experience trying to do this work part-time, we believe it is important to get a starting point that is driven by full-time, grant-backed research. We set our own research approach and encourage those who are interested in the issues to give feedback on it; that, not asking that our process be designed from scratch by our readers, is the intent of these posts.

Note: a positive effect of recent events was a set of substantive concerns about our model raised by non-profit insiders and others. We put all our reasoning and assumptions on our website precisely because we value critiques that will help us improve, and we look forward to responding to and discussing all concerns shortly. This post, however, focuses on the work we’ve been doing recently, evaluations for Cause 4 (K-12 education).

Three of our finalists focus on creating new schools. Replications, Inc. does so through the public school system, while the Knowledge is Power Program (KIPP) and Achievement First create charter schools. In evaluating these programs, we’re hesitant to focus too much on test scores – the link between test scores and useful skills, or later life outcomes, is far from clear – but test scores are a start, and both the media coverage and the organizations themselves have focused on test scores as the primary evidence of their value-added. So what does the evidence say?

So far, it’s very unclear. All three of the organizations discussed sent us data from an incomplete set of their schools; when it came to providing comparison-school data (to put it in context), what we were provided is even more incomplete. The same is true for the independent reports available on the organizations’ websites. For example, the independent reports that KIPP lists focus either on very few schools, or examine many schools but look only at one year. It’s often unclear to us why particular schools, grades, and years were examined and others weren’t, especially in light of the fact that nearly all test-score data is publicly available through state governments. (Collecting such data may be time-consuming, and we wouldn’t necessarily expect the organizations themselves to do so – but what about funders?)

In our minds, if we want to gauge these organizations’ impact, it’s necessary to collect all easily available and relatively recent data (to avoid concerns about cherry-picking particularly successful schools), and study not just the schools relative to district- or state-wide averages, but the trajectory of students’ scores. The question has been raised whether KIPP, for example, effectively selects stronger students for participation, both through its acceptance process and through attrition (students’ dropping out as time goes on). So even if their students outperform “comparison group” students, this isn’t necessarily an effect of the schools themselves; if we can establish, on the other hand, that their students enter at similar levels to their peers but improve gradually over time, the “KIPP effect” will become much clearer.

This sort of observation wouldn’t support a “KIPP effect” as strongly as a true randomized study would, but it would still be much stronger than what we’ve seen to date. We’ve seen comparisons of particular schools to nearby schools, and to statewide averages, but nothing that is either comprehensive or attempts systematically to address the issues discussed above. For example, this report, by the Education Policy Institute which looks across many KIPP schools, but only at 5th grade test scores in 2003-04. Other available reports look only at a few schools, or schools in a particular city for a one academic year.

So, at this point our plan is to bite the bullet: get all the data we can from state governments, and do our own gruntwork putting it in a form that can be systematically analyzed. We certainly don’t want to reinvent the wheel if someone has already done this work – we’d rather read research from experts that have spent much more time on this than we have – but at this point we don’t see an alternative, because we can’t find anyone who is publishing research on the effect of these programs in a way that addresses concerns about selection/attrition/publication bias.

What do you think? Are we going too far? Is there a faster and better way to answer these questions? Do you know of any research that already exists on these issues (for these charities)?