We appreciate the feedback we got on our last blog post. This post will address the substantive issues raised by commenters.

Evaluation and test scores

We aren’t seeking to settle the question of how to measure success in education. That would be biting off far more than we can chew. Instead, we have largely invited applicants to make their case in their own terms, and we aim to grant the one whose case we find the most convincing. This is an unavoidable choice for any donor; our goal is not to seek an unrealistic level of certainty, but simply to put far more time and effort into it than a typical donor can. That said, here are our thoughts on what we find compelling as a measure of success:

Ideally, we’d like to evaluate these organizations based on how well they impact the long-term, life outcomes for the students they serve – this would involve measures such as earnings, criminal record, or maintaining health insurance. But based on general conversations with highly experienced people in the fields of education-related grantmaking and research (including a professor at Harvard’s Graduate School of Education and an experienced program officer at a relevant foundation); conversations with our applicants; and the 1-2 days we spent digging through NCES, IES, and JSTOR/Google Scholar/etc., we don’t believe we’re going to be able to connect K-12 programs to these sorts of outcomes, even loosely, because this type of evaluation does not exist. We also don’t believe we’re going to be able to connect them to college performance, matriculation and graduation, etc., at least in the case of the particular applicants we’ve been discussing, because we’ve asked for this information and they have not provided it.

However, we have seen these applicants point to test scores as evidence for effectiveness, and while this measure leaves a lot to be desired, we still think it’s worth strong consideration. As many have pointed out, improved test scores could be indicative of “teaching to the test” rather than a better education, but given that we are trying to help disadvantaged students who are frequently deficient in basic math and reading skills, we’d guess that improving their ability to do well on reading and math tests more likely than not corresponds with improving their education and prospects. We would guess that improved classroom management (i.e., keeping students attentive – something we believe is a major issue, based on conversations with teachers who work in inner-city public schools), which likely indicates a better classroom environment for education, would also correspond, more likely than not, with improved test scores.

The bottom line is that while this measure leaves a lot unanswered, there are many reasons to see it as meaningful – yet the research we’ve seen does not address specific questions that we feel could be addressed with the information available. We are specifically referring to the questions of (a) whether students are systematically entering these schools with higher performance to begin with; (b) how much of the effect on test scores can plausibly be attributed to attrition. We believe both of these questions could be better (though still far from perfectly) answered through the process outlined before, i.e., examining:

• The difference in the test scores of our applicants’ students versus students in schools in the same area.
• The difference in the change in test scores of our applicants’ students versus students in schools in the same area – particularly for the year in which students enter the schools in question.
• The trajectory of the change in test scores. (I.e., does it all happen in one year or is it consistent over multiple years)
• What proportion of students apply to the lottery, and what proportion leave each year, to get a sense of how plausible these factors are in explaining any difference in test scores.

More specific questions

Michelle: thanks for your recommendations of sites to look at. These are our thoughts:

1. We looked at ERIC several months ago for contextual research on educational evaluation, but should have revisited it for papers on these applicants specifically. We’ve now gone back and found a few papers on KIPP (including this one that looks particularly interesting), and we’ll make sure to look at all of them as we do this research. This was a very helpful suggestion, and we appreciate it.
2. http://ies.ed.gov/ncee/ – we’ve looked before at what seem to us to be the most relevant areas in this resource. Specifically, I’ve looked at the What Works Clearinghouse: A central and trusted source of scientific evidence for what works in education and the Evaluation Studies of the National Center for Evaluation. Both areas have a great deal of information on education but none that deals directly with either our applicants or their methods.
3. http://www.greatschools.net/ – we looked at this earlier in the year and didn’t find it to have the kind of information we’re looking for. It offers parent reviews (usually only a handful, if that); a “Great Schools rating” (which compares a school’s test scores to the the state average; and a link to the test scores available on the state websites. Is there something else here that we missed?

Matt, Andrea, Crystal:

We believe that community-driven research and grantmaking is an idea that has potential, and there are projects (such as Great Nonprofits) that are actively working on it. However, this approach carries an additional set of challenges that we don’t feel equipped to take on. Based on our own experience trying to do this work part-time, we believe it is important to get a starting point that is driven by full-time, grant-backed research. We set our own research approach and encourage those who are interested in the issues to give feedback on it; that, not asking that our process be designed from scratch by our readers, is the intent of these posts.

Note: a positive effect of recent events was a set of substantive concerns about our model raised by non-profit insiders and others. We put all our reasoning and assumptions on our website precisely because we value critiques that will help us improve, and we look forward to responding to and discussing all concerns shortly. This post, however, focuses on the work we’ve been doing recently, evaluations for Cause 4 (K-12 education).

Three of our finalists focus on creating new schools. Replications, Inc. does so through the public school system, while the Knowledge is Power Program (KIPP) and Achievement First create charter schools. In evaluating these programs, we’re hesitant to focus too much on test scores – the link between test scores and useful skills, or later life outcomes, is far from clear – but test scores are a start, and both the media coverage and the organizations themselves have focused on test scores as the primary evidence of their value-added. So what does the evidence say?

So far, it’s very unclear. All three of the organizations discussed sent us data from an incomplete set of their schools; when it came to providing comparison-school data (to put it in context), what we were provided is even more incomplete. The same is true for the independent reports available on the organizations’ websites. For example, the independent reports that KIPP lists focus either on very few schools, or examine many schools but look only at one year. It’s often unclear to us why particular schools, grades, and years were examined and others weren’t, especially in light of the fact that nearly all test-score data is publicly available through state governments. (Collecting such data may be time-consuming, and we wouldn’t necessarily expect the organizations themselves to do so – but what about funders?)

In our minds, if we want to gauge these organizations’ impact, it’s necessary to collect all easily available and relatively recent data (to avoid concerns about cherry-picking particularly successful schools), and study not just the schools relative to district- or state-wide averages, but the trajectory of students’ scores. The question has been raised whether KIPP, for example, effectively selects stronger students for participation, both through its acceptance process and through attrition (students’ dropping out as time goes on). So even if their students outperform “comparison group” students, this isn’t necessarily an effect of the schools themselves; if we can establish, on the other hand, that their students enter at similar levels to their peers but improve gradually over time, the “KIPP effect” will become much clearer.

This sort of observation wouldn’t support a “KIPP effect” as strongly as a true randomized study would, but it would still be much stronger than what we’ve seen to date. We’ve seen comparisons of particular schools to nearby schools, and to statewide averages, but nothing that is either comprehensive or attempts systematically to address the issues discussed above. For example, this report, by the Education Policy Institute which looks across many KIPP schools, but only at 5th grade test scores in 2003-04. Other available reports look only at a few schools, or schools in a particular city for a one academic year.

So, at this point our plan is to bite the bullet: get all the data we can from state governments, and do our own gruntwork putting it in a form that can be systematically analyzed. We certainly don’t want to reinvent the wheel if someone has already done this work – we’d rather read research from experts that have spent much more time on this than we have – but at this point we don’t see an alternative, because we can’t find anyone who is publishing research on the effect of these programs in a way that addresses concerns about selection/attrition/publication bias.

What do you think? Are we going too far? Is there a faster and better way to answer these questions? Do you know of any research that already exists on these issues (for these charities)?

On Monday, Elie Hassenfeld communicated to the board that he posted a comment on December 31, 2007 related to GiveWell that was not under his own name. The comment was made under the name Talia and is linked here. In response to the actions by Mr. Hassenfeld which were inconsistent with GiveWell’s core values of transparency and honesty, the board has decided to impose a financial penalty of $5000. We have updated our ‘Statement from the GiveWell Board of Directors’ to reflect this situation (please note the addition of paragraph 5). We believe that Mr. Hassenfeld’s previous commitment to the goals of GiveWell demonstrates that he can continue to make a positive contribution to the work of the organization in the future in his current role as a program officer.

The following is a statement from the GiveWell Board of Directors. Audio and materials from the meeting are located here. Additional materials related to this statement will be posted soon.

————-

The board has amended its original statement as of January 11, 2008 at 830pm est. Please note the addition of paragraph 5.

———————–

The Board of Directors wishes to emphasize that GiveWell was, is, and will remain committed to the goal of helping those in need by performing and publishing high quality research on charities within selected spheres for free and discretionary use by all interested parties. We are committed to the goals of improving philanthropy and increasing transparency in the sector by publicly sharing our process, our criteria, our methodologies, our deliberations, and our findings. Furthermore, we remain committed to engaging in public dialogue about how best to achieve these goals and encourage continued comment about our organization’s work.

The Board believes that the acts of misrepresentation that were committed are indefensible and are in direct conflict with the goals of the organization, and we condemn them in the strongest possible terms.

The board apologizes for any violations of public trust resulting from Mr. Karnofsky’s actions.

Effective January 3, 2008, Holden Karnofsky has been removed from his position as Executive Director of GiveWell and from his position as Board Secretary. The Executive Director position is now vacant; Tim Ogden will serve as interim Board Secretary. In addition to being removed from his positions, a financial penalty has been imposed on Mr. Karnofsky. While we are removing him from the Executive Director position, we believe that his previous contributions outside of the acts noted above have demonstrated a commitment to the goals of the organization and have been important to accomplishing GiveWell’s work. As a result, Holden will be moved to a Program Officer position, where we believe he will be valuable in helping GiveWell meet its outstanding commitments to applicants and donors. He will also participate in a program of professional development and mentoring.

In response to the actions by Elie Hassenfeld which were inconsistent with GiveWell’s core values of transparency and honesty, the board has decided to impose a financial penalty. We believe that Mr. Hassenfeld’s previous commitment to the goals of GiveWell demonstrates that he can continue to make a positive contribution to the work of the organization in the future in his current role as a program officer.

The GiveWell Board of Directors will begin meeting bi-weekly to develop and approve operating plans for GiveWell and to discuss the organization’s strategy and future. We will institute an organization-wide communications policy that will mandate that all staff and members of the Board of Directors explicitly disclose their affiliation with the organization when communicating with others. The policy will be made public once it has been completed.

We would also like to make clear to all donors to GiveWell since December 1 that we will contact them directly about these recent changes and will provide them the opportunity to have their gift returned if they so choose.

Signed,

Bob Elliott, Chairman
Greg Jensen, Treasurer
Tim Ogden, Interim Secretary
Lucy Bernholz
Virginia Ford

———–

In addition to the above statement from the Board of Directors as a whole, we would like to separately add the following about Holden’s actions, given our greater personal experience with him than other board members’.

We would like to make clear that the actions that Holden took to conceal his identity were improper and indefensible, as were his attempts to ameliorate the situation by offering a financial contribution.

While in this situation Holden acted improperly, we would like to emphasize that we believe Holden’s commitment to the GiveWell cause is genuine. Those who have commented that his involvement in GiveWell is an attempt to run a scam on the public or to gain financially are simply wrong. His substantial sacrifices financially and personally to take GiveWell from an idea to a reality demonstrate his passion for the values of the organization.

In addition, we would like to emphasize that we believe and have seen in action his stated commitment to the values of openness and honesty. While there are many incidents where Holden did not clearly identify his affiliation with the organization, we view these actions to be the result of his core mistake of thinking improperly about how to represent himself when communicating online.

In our previous professional and personal experiences with Holden, he has shown directness and honesty with those around him, and demanded the same from others. He has acknowledged weaknesses and worked with sincerity to improve those limitations. And, in the past when he has realized that his behavior or thinking was wrong, he not only been willing to change his mind and accept explicit personal responsibility, but take steps to ensure that similar action does not happen again.

It is clear to us that he has realized that his actions were improper, taken responsibility for his actions, and is committed to making sure that he communicates in a way that holds true to the values of honesty and transparency moving forward. We know Holden is committed to these values and expect that he will demonstrate to the community that this behavior was not consistent with his character through his continued work at GiveWell.

We recognize that people can make mistakes and that they can learn powerful lessons from those failures. When we think about the whole picture of Holden, and view his mistakes in the context of his contribution to GiveWell so far as well as his character prior to this project, we believe that he will continue to make a positive impact for the organization in the future.

Bob Elliott, Chairman
Greg Jensen, Treasurer

I did what is apparently known as “astroturfing,” using a fake ID to try to get attention for GiveWell. Why did I do this?

1. I had a horrible lapse in judgment.

2. I have been low on sleep, and running around like a madman. I’d like to think that had something to do with it, but if you want to dismiss that and just call me an idiot, I can’t argue with you.

• I used an old Ask Metafilter account to pose a question about where to find a good charity (the kind of question I was asking everyone I knew a year ago), then I created a new account as Holden0 to answer it with GiveWell (I didn’t intend to hide the fact that the GiveWell link itself came from me, the founder of the project – but posing the setup question as an outsider was completely inappropriate).
• I made two blog comments and sent ten emails from a handle that did not identify me.
• I also made numerous other comments from a handle that did identify me (“Holden”), but did not put up front that I was the founder of the project.

The gory details are here. The current status of the situation is here.

This was a horrible lapse in judgment. I was simply trying to make comments that would not be immediately dismissed as “plugs,” but rather explored on their merits. I now recognize that by not disclosing my identity, I created the appearance of an “objective” observer, and that was wrong.

I hope as you read this, you can identify with having a horrible lapse in judgment. Everything that is true of me is something that has been untrue of me in a moment of weakness, and that includes the value of honesty, which remains the most important one. I hope you can recognize that the best thing I can do is admit it immediately, apologize, and not try to hide anything. That is what I am doing. I hope you still believe in me as someone who will admit to his lapses and mistakes, if not someone who can be counted on to be free of them.

Transparency is the one thing about GiveWell that everyone seems to like. Our focus on measurement is much more contested. I believe that the connection is tight, though, because both are necessary consequences of humility, which is probably the last word you ever thought I’d ask to be associated with.

Transparency is a really big deal to us because we believe that no matter how much we learn and no matter how hard we work, we can always be wrong. That’s why we invite as many people as possible into the conversation.

(When I look at large foundations making multimillion-dollar decisions while keeping their data and reasoning “confidential” – all I see is a gigantic pile of the most unbelievably mind-blowing arrogance of all time. I’m serious. Deciding where to give is too hard and too complex – with all the judgment calls and all the different kinds of thinking it involves, there is just no way Bill Gates wouldn’t benefit from having more outside perspectives. I don’t care how smart he is.)

Measurement is about inviting someone else into the conversation: The Facts. The Facts have a lot to say, and they often contradict what we would have thought. That’s why we have to listen to them. Like transparency, measurement takes a lot of extra effort and expense; like transparency, it can’t solve all your problems by itself; and like transparency, it’s easily worth it if you agree that the issues are extremely complex, and that no matter how much sense something makes in your head, The Facts might disagree.

(And yes, when I hear people talking about how they’re “too busy helping people” to measure, I hear arrogance there too. The only way you could ever take that position is if you are so sure of what you’re doing that you think learning more about it isn’t worth the 10% more you can spend doing the same untested thing. How high on your own infallibility do you have to be to come to that conclusion?)

To a lot of people, humility means speaking with a certain tone of voice, or just plain keeping your mouth shut. If that’s what you think humility is, we don’t have it. To us, humility is constantly saying “The things that make sense to me could be wrong – that’s why I’m going to do everything I can to test them, against others’ ideas and against reality.” Instead of being silently dissatisfied with charity, we’re loudly dissatisfied, so that anyone who disagrees can respond. Instead of happily assuming our dollars are doing good, we demand to see The Facts, so they can respond too.

Laugh if you must, but in the end humility is the defining value of the GiveWell project. We’re not here to impose our solutions on the sector; we’re here because we want to see more questions.